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9^'^  ANNUAL  TECHNOLOGY  CONFERENCE 


"Investigation,  Measures,  and  Lessons  Learned  About  the  Reiationship  Between  CM  MI  Process 

Capabiiity  and  Project  or  Program  Performance" 

Denver,  CO 

16  -  19  November  2009 


Agenda 


Tuesday^,  November  17^  2009 


EXECUTIVE  PANEL 

Moderator:  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon  Company 

•  Mr.  Mike  Twyman,  VP  of  the  Integrated  Command  and  Control  Business  Unit,  Northrop  Grumman  Corp. 

•  Mr.  David  J.  Tyler,  Sr.  Manager,  IIS  Enterprise  Process  Effectiveness,  Raytheon  Company 

•  Mr.  Wesley  Covell,  President  of  Defense  Programs,  Harris  Corp. 

•  Ms.  Lynn  Penn,  Director  of  Process  Management,  Lockheed  Martin  Corporation 

•  Mr.  Girish  Seshagiri,  CEO,  Advanced  Information  Services,  Inc. 

LUNCHEON  SPEAKER: 

•  Mr.  Hal  Wilson,  Director,  Engineering  Defense  Systems  Division,  Northrop  Grumman  Information  Systems 

Concurrent  Sessions 

TRACK  1-  GRAND  MESA  D/E-  CMMI  AND  PROCESS  IMPROVEMENT 
Session  Chair:  Mr.  Jack  Ferguson,  Software  Engineering  Institute 

•  9386  -  CMMI  for  Large  Scale/Systems  of  Systems  Engineering  Projects,  Mr.  Patrick  McCusker,  Booz  Allen  Hamilton 

•  Making  the  CMMI  Sing  -  A  Framework  for  Performance  Excellence,  Mr.  Jeff  Dutton,  Jacobs  Technology,  Inc 

•  93 12  -  CMMI  in  a  Small  Company:  The  Cobbler’s  Children  Can  Have  Shoes  ( And  Best  Practices  ,  Mr.  Michael  Knox,  TECHSOFT,  Inc 
TRACK  2  -  GRAND  MESA  F  -  PRACTICAL  GUIDANCE 

Session  Chair:  Mr.  Gene  Miluk,  Software  Engineering  Institute 

•  9179  -  Work  On  Your  Engineering  Business,  Not  In  It,  Mr.  Rolf  Reitzig,  Cognence,  Inc 

•  Process-Performance  Base  Reliability,  Mr.  William  Winkel,  Northrop  Grumman  Corporation 

•  9391  -  Choices  to  be  Made  in  CMMI  Adoption,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  9298  -  Assurance  for  CMMI:  A  Toolbox  for  Multiple  Cyber  Challenges,  Mrs.  Michele  Moss,  Booz  Allen  Hamilton 
TRACK  3  -  HIGHLANDS-  CMMI  ECONOMICS  &  BUSINESS  VALUE 

Session  Chair:  Mr.  Geoff  Draper,  Harris  Corporation  &  Mr.  Bob  Ferguson,  Software  Engineering  Institute 

•  9147  -  Dynamic  Program  Schedule,  Cost  and  Returns  Analysis,  Mr.  Phillip  Fahringer,  Lockheed  Martin  Corporation 

•  9 1 84  -  The  Economics  of  CMMI,  Mr.  Mick  Campo,  Raytheon  Company 

•  9185  -  CMMI  Economics  101:  CMMI  for  Executives,  Mr.  Geoff  Draper,  Harris  Corporation 

TRACK  4  -  CHASM  CREEK  -  HIGH  MATURITY 

Session  Chair:  Mr.  Dennis  Goldensen,  Software  Engineering  Institute  &  Mr.  Fred  Schenker,  Software  Engineering  Institute 

•  9146  -  Goal  Question  -  Model,  Mr.  Michael  Campo,  Raytheon  Company 

•  9389  -  Marking  CMMI  Level  5  Statistical  Principles  Palatable  to  an  Employee-Wide  Demographic,  Ms.  Deepti  Sharma,  OST 

•  91 16  -  Changing  Behavior:  The  Key  to  Adoption  Complex  Process  Technology,  Dr.  Gene  Miluk,  SEI 

•  9401  -  Achieving  Quality  QPPO  via  Effective  Usage  of  PPBs  and  PPMs,  Dr.  Bin  Cong,  CRS 

TRACK  5  -  MESA  VERDE  -  CMMI  GOVERNMENT  &  ACQUISITION 

Session  Chair:  Ms.  Lorraine  Adams,  Software  Engineering  Institute  &  Mr.  Mike  Phillips,  Software  Engineering  Institute 
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•  8806  -  Benefits  to  the  Evolution  of  High  Maturity  Software  Development:  A  15  Year  Case  Study,  Mr.  Daniel  Drew,  United  Space  Alliance 

•  9306  -  Directive  Documents  and  ITAR  Made  Easy,  Mr.  Kenneth  Weinberg,  Raytheon  Corporation 

•  8907  -  How  CMMI  was  Used  for  Process  Improvement  in  the  Support  of  Government- Wide  Acquisition  Contract  (GW AC)  Vehicles,  Mrs. 
Sharon  Cobb,  Flanagan,  SAIC 

•  9403  -  Tailoring  CMMI  for  an  Enterprise  Resource  Planning  COTS  Software  Environment,  Ms.  Alison  L.  Schwier,  U.S.  Army 
TRACK  6  -  WIND  RIVER  -  APPRAISALS 

Session  Chair:  Mr.  Ken  Nidiffer,  Software  Engineering  Institute  &  Mr.  Kenneth  Weinberg,  Raytheon  Corporation 

•  9398  -  Hocus  Pocus,  What’s  With  All  the  Issues  About  Non-Focus?,  Mr.  Paul  Byrnes,  Integrated  Systems  Diagnostics,  Inc. 

•  9136  -  Making  the  Most  of  GP3.2,  Ms  Susan  Byrnes,  Natural  SPI,  Inc 

•  9130  -  Supporting  the  High  Maturity  Process  Improvement  and  Understanding  the  Application  SCAMPISM  Method  To  It.,  Mr.  Kobi  Vidar, 
K.V.P.  Consulting 

•  9385  -  Reducing  the  Cost  and  Increasing  the  Value  of  CMMI  Re- Appraisals,  Mrs.  Beth  Layman,  Layman  and  Layman 
TRACK  7  -  WIND  STAR  -  CMMI  V1.3  TOPIC  &  CMMI-SVCS,  LEAN 

Session  Chair:  Ms.  Susan  Bassham,  U.S.  Army  Aviation  &  Missile  Command 

•  CMMI  VI. 3  -  From  the  Past  to  the  Future,  Mr.  Mike  Philllips,  Software  Engineering  Institute 

•  9167  -  Lessons  Learned  Piloting  the  CMMI  for  Services,  Ms.  Diane  Mizukami  (Williams),  Northrop  Grumman  Information  Systems 

•  9126  -  CMMI  for  Services:  An  Approach  to  Improve  Your  Program  Management  Office,  Ms.  Patricia  Mitryk,  Cognence,  Inc 

•  9299  -  Creatively  Apply  CMMI  SVC  in  a  Very  Small  Consulting  Firm,  Mr.  Bill  Smith,  Leading  Edge  Process  Consultants 


Wednesday^  November  18^  2009 


Concurrent  Sessions 

TRACK  1  -  GRAND  MESA  D/E  -  CMMI  AND  PROCESS  IMPROVEMENT 
Session  Chair:  Mr.  Jack  Ferguson,  Software  Engineering  Institute 

•  9304  -  Sustainment  and  Continued  Institutionalization  of  Best  Practices  and  CMMI  at  SPAWAR,  Mr.  Michael  Kutch,  Space  &  Naval  Warfare 
Systems  Center  -  Atlantic 

•  9178  -  CMMI  Process  Improvement,  Its  not  a  technical  Problem,  It’s  a  People  Problem!,  Mr.  Rolf  Reitzig,  Cognence,  Inc 

•  9106  -  The  Uses  of  the  Peer  Review  beyond  CMMI,  Mr.  Paul  Nugent,  General  Dynamics  Corporation 

•  9246  -  Integrating  Corporate  Goals  and  Processes  Using  the  Engineering  Lifecycle  Vee  Model,  Dr.  Keven  Forsberg,  The  Center  for  Systems 
Management 

•  9379  -  NAV air’s  Process  Asset  Library  (PAL),  A  Step  Toward  A  Corporate  Organizational  Set  of  Standard  Processes  (OSSP),  Ms.  Judy 
Overhauser-Duett,  NAV  AIR 

•  9153  -  After  13  years,  I  have  learned  ....  Tools  do  not  solve  your  problems,  Mr.  John  Bethmann,  Concurrent  Technologies  Corporation. 

•  9387  -  10.5  Process  Improvement,  Mistakes  From  Top  Executives?,  Mr.  Carlos  Caram,  CSD  Brasil 

•  9144  -  Transitioning  From  a  CMMI  Implementer  to  an  Appraiser,  Mr.  Warren  Scheinin,  Northrop  Grumman  Corporation 
TRACK  2  -  GRAND  MESA  F  -PRACTICAL  GUIDANCE 

Session  Chair:  Mr.  Gene  Miluk,  Software  Engineering  Institute 

•  9138  -  High  Velocity  Performance  Improvement,  Mr.  Jeff  Dutton  Jacobs  Technology,  Inc 

•  9202  -  Statistical  Tune-Up  of  the  Peer  Review  Process,  Mr.  Tom  Lienhard,  Raytheon  Missile  Systems 

•  Everything  You  Wanted  to  Know  About  CMMI  and  Six  Sigma  but  Did  Not  Know  Who  to  Ask,  Tom  Lienhard,  Raytheon  Missile  System 

•  9275  -  CMMI®  in  the  Social  Media  (For  the  Social  Media-Challenged!),  Mr.  Bill  Smith,  Leading  Edge  Process  Consultants 

•  9214  -  Hi,  my  name  is  Root  Cause  Analysis.  Have  we  met?,  Mr.  Craig  Hale,  Esterline  Control  Systems  -  AVISTA 

•  8787  -  Improving  Process  Institutionalization  Through  Process  Training,  Ms.  Ellen  Chilikas,  Raytheon  Company 

•  9291  -  “You  Say  Tomato,  I  Say  Eggplant:  Comparing  Process  References  for  Systems  Engineers  and  Project  Managers  in  a  CMMI®- 
Compliant  Organization”,  Mr.  Peter  Henry,  BAE  Systems 

•  9354  -  Strategies  for  Process  Definition  and  Deployment  Part  1,  Mr.  Fred  Schenker,  SEI 

•  Strategies  for  Process  Definition  and  Deployment  Part  2  ,  Mr.  Fred  Schenker,  SEI 

•  Shrinking  the  Elephant:  If  Implementing  CMMI  Practices  Looks  Like  More  Effort  than  it’s  Worth,  Let’/s  Look  Again,  Sam  Fogle,  ACE 
Guides,  EEC 

TRACK  3  -  HIGHLANDS  -  CMMI®  ECONOMICS  &  BUSINESS  VALUE 

Session  Chair:  Mr.  Geoff  Draper,  Harris  Corporation  &  Mr.  Bob  Ferguson,  Software  Engineering  Institute 

•  9213  -  QPMing  Your  SEPG,  Mr.  Craig  Hale,  Esterline  Control  Systems  -  AVISTA 

•  9223  -  We’re  Already  There:  Matching  Existing  High  Maturity  Behaviors  to  the  CMMI®  Model,  Mr.  Bradley  Bittorf  Raytheon  Company 

•  9190  -  CMMI®  Economics  501 :  High  Maturity,  Mr.  Mike  Campo,  Raytheon  Company 

•  9378  -  Using  Corporate  Finance  Principles  to  Easily  Determine  Return  on  Investment  (ROI),  Ms.  Deepti  Sharma,  OST 

•  8909  -  Consistency  in  Quality  Assessments,  Mrs.  Debra  Perry,  Harris  Corporation 

•  8871  -  MSI  Execution:  Change  Happens,  How  to  Deal  with  It,  Ms.  Jill  Brooks,  Raytheon  Company 

•  9177  -  Lessons  Learned  Using  Earned  Value  Management  System  to  Track  Effort  and  Schedule  Weekly  at  the  Individual  and  Team  Level  and 
Be  Able  to  Detect  a  One-Day  Schedule  Slip,  Mr.  Girish  Seshagiri,  Advanced  Information  Services  Inc. 

•  9188  -  CMMI  Economics  203:  Model  Tailoring,  Mr.  Jeff  Dutton,  Jacob  Technologies,  Inc 

TRACK  4  -  CHASM  CREEK  -  HIGH  MATURITY 

Session  Chair:  Mr.  Dennis  Goldenson,  Software  Engineering  Institute  &  Mr.  Fred  Schenker,  Software  Engineering  Institute 

•  9217  -  A  Taxonomy  of  CMMI®  High  Maturity  Performance  Models,  Dr.  Richard  Welch,  Northrop  Grumman  Corporation 
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•  9245  -  The  Selection  and  Deployment  of  a  Standard  COTS  Monte  Carlo  Software  Tool,  Mr.  Fred  Oleson,  BAE  Systems,  Inc. 

•  9232  -  Piloting  a  Hybrid  Requirements  Engineering  Process  for  Translating  Qualitative  Information  into  Quantitative  Performance  Measures, 
Mr.  Dennis  Goldenson,  SE 

•  9168  -  How  I  Created  Our  Peer  Review  Baselines  and  Models,  Ms.  Diane  Mizukami  (Williams),  Northrop  Grumman  Information  Systems 

•  9216  -  ABCs  of  Process  Performance  Models,  Dr.  Richard  Welch,  Northrop  Grumman  Corporation 

•  9294  -  Using  Hidden  Markov  Models  as  a  Statistical  Process  Control  Technique:  An  Example  from  a  ME  5  Organization,  Mr.  Robert  Moore, 
Business  Transformation  Institute,  Inc. 

•  9407  -  Multi- Attribute  Modeling  and  Practical  Use,  Mr.  David  Sobetski,  General  Dynamics  Land  Systems 

•  9390  -  Discrete  Event  Simulation  for  QPM  -  Can  it  really  be  that  easy?,  Ms.  Deepti  Sharma,  OST 

TRACK  5  -MESA  VERDE  -  CMMI®  GOVERNMENT  &  ACQUISITION 

Session  Chair:  Ms.  Lorraine  Adams,  Software  Engineering  Institute  &  Mr.  Mike  Phillips,  Software  Engineering  Institute 

•  0000  -  Using  CMMI®  for  Acquisition  in  Integration  Organizations,  Mr.  Steve  Kelley,  Northrop  Grumman  Corporation 

•  8741  -  Leveraging  CMMI®  for  Acquisition  to  Improve  Organizational  Workforce  Performance,  Dr.  Kenneth  Nidiffer,  SEI 

•  9301  -  Improving  Processes  the  NSA  Way,  Mr.  Robert  Moore,  Business  Transformation  Institute,  Inc 

•  91 14  -  Implementing  Requirements  Management  To  Deliver  Life  Cycle  Software  Solutions  That  Ensure  Warfighting  Superiority  and 
Information  Dominance:  How  We  Moved  The  Rock,  Mr.  Harlan  Black,  U.S.  Army  CECOM 

•  8764  -  Predicting  Quality,  Mr.  Donald  Beckett,  Quantitative  Software  Mgt. 

•  9326  -  “Patience  is  a  Virtue”-  A  Day  in  the  Life  of  a  SEPG  Lead,  Ms.  Tammye  Thornton,  Department  of  Navy 

•  9321  -  Enterprise  Architecting  and  the  Incorporation  of  Early  Systems  Engineering  Data  Into  the  Leadership  Decision  Making  Process  During 
Concept  Development,  Mr.  George  Freeman,  United  States  Air  Force 

•  9154  -  Simplifying  Lifecycle  Definition  Process,  Mr.  John  Bethmann;  Concurrent  Technologies  ,  Corp 

•  CMMI  Economics  203:  Model  Tailoring,  Mr.  Jeffrey  L.  Dutton,  NDIA  Systems  Engineering  Division 

TRACK  6  -  WIND  RIVER  -  APPRAISALS 

Session  Chair:  Mr.  Ken  Nidiffer,  Software  Engineering  Institute  &  Mr.  Ken  Weinberg,  Raytheon  Corporation 

•  9371  -  Appraisals  and  CMMI®  Gotchas  -  Lessons  in  CMMI®  Use  and  Appraisal  Preparation,  Mr.  Neil  Potter,  The  Process  Group 

•  93 1 1  -  Piloting  Results-Based  Appraisals,  Mr.  Larry  McCarthy  Global  Software  Group 

•  9129  -  Using  the  SCAMPI  Method  to  lead  CMMI®  Multi-Constellations  with  Additional  Standards  Progress  Check  and  Appraisal,  Mr.  Kobi 
Vidar,  K.V.P.  Consulting 

•  9234  -  A  View  from  the  Trenches:  Practical  Guidance  for  Appraisal  Artifact  Management,  Mr.  David  Dickinson,  Northrop  Grummon 
Corporation 

•  9287  -  How  to  Prepare  for  a  CMMI®  SCAMPI  A:  Applying  Agile  Concepts  to  Save  Time  and  Money,  Ms.  Michele  Shaw.  Fraunhofer  Center, 
Maryland 

•  9365  -  Streamlining  Processes  and  Appraisals,  Mr.  Gary  Natwick,  Harris  Corporation 

•  9189  -  CMMI®  Economics  202:  Appraisals,  Mr.  Geoff  Draper,  Harris  Corporation 

•  9227  -  The  ABC’s  of  Class  C,  Class  B,  Class  A’s:  Metrics  and  Lessons  Learned  from  Appraisal  Event  Scheduling,  Mr.  Bradley  Bittorf, 
Raytheon  Company 

TRACK  7  -  WIND  STAR-  CMMI®  -  SVCS,  LEAN,  SMSETTINGS,  ETC. 

Session  Chair:  Ms.  Susan  Bassham,  U.S.  Army  Aviation  &  Missile  Command 

•  9396  -  Applying  the  CMMI®  for  Services  to  the  Process  Group  (Physician,  Heal  Thyself!),  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  9274  -  An  Overview  of  CMMI®-SVC  for  CMMI®-DEV  Enthusiasts,  Mr.  Bill  Smith,  Leading  Edge  Process  Consultants 

•  9137  -  ITIL  V3.0  Compliance  Benchmarking  with  CMMI-SVC  SCAMPI  A,  Mr.  Jeff  Dutton,  Jacobs  Technology,  Inc. 

•  9397  -  Strategies  for  Transitioning  to  CMMI-SVC,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  9196  -  Interpretation  and  Lesson  Learned  from  High  Maturity  Implementation  of  CMMI-SVC,  Mr.  Kobi  Picker,  K.V.P.  Consulting 

•  9264  -  Applying  Lean  Principles  to  the  CMMI®  for  Services  and  ITIL,  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

•  9203  -  How  Rocket  Scientist  Implement  High  Maturity,  Mr.  Tom  Lienhard,  Raytheon  Missile  Systems 


Thursday^.  November  19^.  2009 

NDIA  Systems  Engineering  Division  CMMI  Working  Group 
CMMI  vl.3  Planned  Improvements 

Concurrent  Sessions 

TRACK  1  -  GRAND  MESA  D/E  -  CMMI®  AND  PROCESS  IMPROVEMENT 
Session  Chair:  Mr.  Jack  Ferguson,  Software  Engineering  Institute 

•  93 13  -  Mission  Readiness  Through  Integrated  Decision  Making  . . .  BIM,  BAM  and  POWI,  Mr.  James  Watson, 
Facility  Lifecycle  Group  Div.  of  MACTEC  E  &  C 

•  9165  -  Make  PIIDs  Easy  -  No  Surprises!,  Ms.  Vicki  Galanko,  Lockheed  Martin,  IS&GS-Civil 
TRACK  2  -  GRAND  MESA  F-  PRACTICAL  GUIDANCE 

Session  Chair:  Mr.  Gene  Miluk,  Software  Engineering  Institute 

•  9308-  Post  Merger  Process  Syndrome:  Integrating  &  Refining  Organizational ,  Processes,  Mr.  Michael  Kutch, 
Space  &  Naval  Warfare  Systems  Center  -  Atlantic 
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•  9141  -  Tools  and  Implementation  Strategies  for  Process  Improvement  via  CMMI®  for  Comprehensive  Software 
Lifecycle  Management,  Mrs.  Denise  Padilla,  Sandia  National  Laboratories 

•  9346  -  Exploiting  Decision  to  Requirements  Traceability,  Mr.  John  Fitch,  SAIC 

•  Strategies  for  Process  Definition  and  Deployment  Part  2,  Mr.  Alfred  Schenker,  SEI 

TRACK  3  -  HIGHLANDS  CMMI®  ECONOMICS  &  BUSINESS  VALUE 

Session  Chair:  Mr.  Geoff  Draper,  Harris  Corporation  &  Mr.  Bob  Ferguson,  Software  Engineering  Institute 

•  9181  -  Are  You  Doing  R&D,  or  Catch-up  &  D?  Are  you  Building  Software,  or  Hopeware?,  Mr.  Rolf  Reitzig, 
Cognence,  Inc. 

•  9283  -  CMMI®  Measurement  and  Metrics,  Dr.  Elliot  Lynn,  CECOM  SEC 

•  9324  -  Measuring  True  Agility  in  Agile  Software  Development,  Mr.  Robert  Moore,  Business  Transformation 
Institute,  Inc. 

TRACK  4  -  CHASM  CREEK-  HIGH  MATURITY 

Session  Chair:  Mr.  Dennis  Goldenson,  Software  Engineering  Institute  &  Mr.  Fred  Schenker,  Software  Engineering 
Institute 

•  9143  -  Using  Moving  Average  Models  to  Predict  Process  Performance,  Mr.  Robert  Tuthill,  Northrop  Grumman 
Corporation 

•  9148  -  Use  of  Monte  Carlo  Simulation  for  a  Peer  Review  Process  Performance  Model,  Ms.  Emerald  Russo,  BAE 
Systems 

•  9163  -  Picking  the  Right  Process  Improvements,  Mr.  Joseph  Vandeville,  Northrop  Grumman  Corporation 

•  9244  -  Perspectives  on  Use  and  Organizational  Impact  of  Measurement  and  Analytical  Methods  in  CMMI® 

High  Maturity  Organizations,  Dr.  Dennis  Goldenson,  SEI 

TRACK  5  -  MESA  VERDE  -  CMMI®  GOVERNMENT  &  ACQUISITION 

Session  Chair:  Ms.  Lorraine  Adams,  Software  Engineering  Institute  &  Mr.  Mike  Phillips,  Software  Engineering  Institute 

•  9359  -  Moving  your  Security,  Business  Continuity,  and  IT  Activities  to  the  Next  Level  with  the  CERT® 
Resiliency  Management  Model,  Ms.  Gibbie  Lu  Hart,  SEI 

•  9366  -  Enjoy  the  Scenery  on  the  Path  to  High  Maturity  Ms.  Susan  Bassham,  U.S.  Army  Aviation  and  Missile 
Command 

•  921 1  -  Transforming  Your  Way  to  Control  Charts  that  Work,  Mr.  Richard  Welch,  Northrop  Grumman 
Corporation 

•  9292  -  Systems  Engineering  Processes  Improvement  using  the  CMMI®  in  large  System  of  Systems  Space 
Programs,  Ms.  Revital  Goldberg,  Israel  Aerospace  Industries 

TRACK  6  -  WIND  RIVER  -  APPRAISALS 

Session  Chair:  Mr.  Ken  Nidiffer,  Software  Engineering  Institute  &  Mr.  Ken  Weinberg,  Raytheon  Corporation 

•  9373  -  Lockheed  Martin  Aeronautics  Appraisal  Project  Management  Strategy,  Ms.  Pam  Hudson,  Lockheed 
Martin  Aeronautics 

•  9369  -  Lockheed  Martin  Aero  Standard  Approach  -  A  Strategy  to  Select  Objective  Evidence  for  the  PHD,  Mrs. 
Pam  Hudson,  Eockheed  Martin  Aeronautics 

•  9187  -  Level  5  the  Hard  Way  -  Persevering  through  Organizational  Changes,  Ms.  Doma  Witkowski,  Lockheed 
Martin  Corporation 

•  9383  -  What?  I  Need  to  Create  an  Appraisal  Database  Containing  Thousands  of  Artifacts!  HELP! . . .  Sensible 
PHD  Strategies,  Mr.  Sam  Fogle,  ACE  Guides,  LLC 

TRACK  7  -  WIND  STAR  MULTI-MODELS 

Session  Chair:  Ms.  Susan  Bassham,  U.S.  Army  Aviation  &  Missile  Command 

•  9257  -  The  Next  Step  in  Process  Evolution:  CMMI®  and  TSP/PSP,  Mr.  Jeffrey  Schwalb,  Naval  Air  Systems 
Command 

Panel  -  The  Next  Step  In  Process  Evolution:  CMMI  and  TSP/PSP 

1 .  Mr.  Jeff  Schwalb,  NAVAIR 

2.  Ms.  Kathy  Smith,  EDS 

3.  Mr.  Girish  Seshagiri,  CEO,  Advanced  Information  Services,  Inc. 

4.  Mr.  Dave  Webb,  Hill  AFB 

•  9204  -  Everything  You  Wanted  to  Know  About  CMMI®  and  Six  Sigma  but  Did  Not  Know  Who  to  Ask,  Mr. 
Tom  Lienhard,  Raytheon  Missile  Systems 

•  How  Rocket  Scientist  Implement  High  Maturity,  Tom  Lienhard,  Raytheon  Missile  Systems 

•  9266  -  Rapidly  Implementing  Lean  CMMI®  Processes  That  Meet  Business  Needs,  Mr.  Tim  Olson,  Lean 

•  Solutions  Institute,  Inc. 

•  9394  -  Comparing  Scrum  and  CMMI®  How  Can  They  Work  Together  Mr.  Neil  Potter,  The  Process  Group 
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CENTER  >  DENVER,  CO 


SPONSORED  BY: 

NATIONAL  DEFENSE  INDUSTRIAL 
ASSOCIATION  SYSTEMS 
ENGINEERING  DIVISION 
IN  CONJUNCTION  WITH: 

^  Software  Engineering  Institute 

Carnegie  Mellon  University 

This  conference  brings  together  the  managers 
and  professionals  involved  in  Acquisition 
Management,  Systems  Engineering,  Program 
Management,  Software  Development,  Process 
Improvement,  Six  Sigma  and  related  activities  for 
the  purpose  of  advancing  the  state-of-the-art  in 
process  improvement  and  achieving  a  higher  state 
of  process  capability  in  engineering  development 
in  order  to  reduce  cost,  schedule  and  risk,  and 
improve  overall  quality. 

Who  Should  Attend? 

Defense,  aerospace  and  commercial  companies, 
CMMI®  Transition  Partners,  Department 
of  Defense  organizations,  small  companies 
specializing  in  software  and  systems  engineering 
development,  tools  and  processes,  acquisition,  or 
services,  and  other  government  agencies. 

What  will  be  presented? 

A  wide  variety  of  presentations,  including  the 
new  CMMI  for  Services,  integrated  process 
improvement,  Lean/Agile  and  Six  Sigma 
approaches,  and  evolving  approaches  and  lessons 
learned  involving  SCAMPISM  appraisal  methods. 
The  latest  state  of  the  CMMI  V.3  release  will  be 
presented  and  questions  answered. 
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ANNOUNCEMENT 

The  National  Defense  Industrial 
Association,  Systems  Engineering 
Division,  in  conjunction  with  the 
Software  Engineering  Institute,  Carnegie 
Mellon  University,  is  pleased  to  announce 
the  9th  Annual  CMMI®  (Capability 
Maturity  Model  Integration)  Technology 
Conference  &C  User  Group.  This  premier 
conference  will  be  held  November  16-19, 
2009,  at  the  Hyatt  Regency  Tech  Center 
in  Denver,  Colorado. 

The  purpose  of  the  conference  is  to 
exchange  ideas,  concepts  and  lessons 
learned  concerning  the  continuing 
evolution,  adoption  and  use  of  the 
CMMI®  and  its  associated  appraisal 
(assessment  and  evaluation)  methods. 

This  conference  brings  together  CMMI® 
adopters,  users,  developers  and  appraisers, 
as  well  as  those  with  general  interest 
in  process  improvement.  It  provides  a 
forum  for  the  free  exchange  of  ideas  and 
affords  a  unique  opportunity  to  meet  with 
the  sponsors,  developers  and  stewards 
of  the  CMMI®,  as  well  as  those  offering 
CMMI®  training  and  implementation 
assistance.  Emphasis  will  be  placed  on 
CMMI®  implementation  methods  and 
strategies,  return  on  investment  and 
program/project  performance  benefits. 

PLAN  TO  STAY  FOR  THE 
THURSDAY  AFTERNOON 
REVIEW  OF  CMMI  VI. 3! 

CONTACTS 

Ms.  Kelly  Seymour,  Meeting  Planner, 
kseymour@ndia.org,  (703)  247-2583 

Conference  Chair;  Mr.  Bob  Rassa, 
Director,  Engineering  Programs, 

Raytheon  Space  &  Airborne  Systems  or 
Raytheon  SAS,  rcrassa@raytheon.com, 
(310)  985-4962 

Technical  Program  Chairs:  Mr.  Jeffrey 
Dutton,  Jacobs  Technology  ITSS,  jeff. 
dutton@jacobs.com,  and  Mr.  Rick 
Barbour,  Software  Engineering  Institute, 
reb@sei.cmu.edu 


CMMI®  TECHNOLOGY  CONFERENCE 
AND  USER  GROUP 

HYAH  REGENCY  TECH  CENTER  ►  DENVER,  COLORADO 
NOVEMBER  16-19, 2009 

The  CMMI®  was  developed  in  cooperation  with  the  Department  of  Defense, 
Industry  and  the  Software  Engineering  Institute,  and  has  become  the  defacto 
standard  for  integrated  process  improvement  across  multiple  disciplines  within 
commercial.  Department  of  Defense  and  government  organizations.  Although 
sponsored  by  the  Department  of  Defense  and  NDIA,  the  CMMI®  is  used 
by  commercial  as  well  as  government  and  industry  organizations,  and  this 
conference  will  address  all  applications.  The  purpose  of  CMMI®  is  to  provide  for 
improvements  in  cost,  schedule  and  overall  performance  of  projects  in  engineering, 
acquisition,  and  services  by  eliminating  “stovepipe”  maturity  models  and  allowing 
organizations  to  integrate  their  process  improvement  efforts.  CMMI®  has  been 
shown  to  reduce  costs,  to  implement  internal  process  improvement,  including 
appraisals  (assessments  &C  evaluations)  and  provide  a  common  baseline  and  lexicon 
for  process  improvement. 

CONFERENCE  OBJECTIVE 

This  conference  brings  together  the  managers  and  professionals  involved  in 
Acquisition  Management,  Systems  Engineering,  Program  Management,  Software 
Development,  Process  Improvement,  Six  Sigma  and  related  activities  for  the  purpose 
of  advancing  the  state-of-the-art  in  process  improvement  and  achieving  a  higher  state 
of  process  capability  in  engineering  development  in  order  to  reduce  cost,  schedule 
and  risk,  and  improve  overall  quality. 

CONFERENCE  AHIRE 

Appropriate  dress  for  this  conference  is  business  for  civilians  (coat  and  tie)  and 
class  A  uniform  or  uniform  of  the  day  for  military. 

CONFERENCE  PROCEEDINGS 

Proceedings  will  be  available  on  the  web  through  the  Defense  Technical  Information 
Center  (DTIC),  and  will  be  available  one  to  two  weeks  after  the  conference.  You 
will  receive  notification  via  e-mail  once  proceedings  are  posted  and  available  on 
the  web 
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SUNDAY,  NOVEMBER  15, 2009 

3:00  pm  -  6:00  pm  Registration  Open 

Located  in  Grand  Mesa  Foyer,  2“^  floor 

MONDAY,  NOVEMBER  16, 2009 


7:00  am  -  6:00  pm 

Registration  Open 

Located  in  Grand  Mesa  Foyer,  2“'-  floor 

7:00  am  -  8:00  am 

Continentai  Breakfast  (Tutoriai  Attendees  Oniy) 

Located  in  Grand  Mesa  Foyer,  2^  floor 

8:00  am  -  5:00  pm 

Tutoriai  Sessions  (Tutoriai  Attendees  Oniy) 

9:45  am  -  10:15  am 

Break  (Tutoriai  Attendees  Oniy) 

Located  in  Atrium  Display  Area,  2‘‘  floor 

12:00  pm  -1:00  pm 

Lunch  (Tutoriai  Attendees  Oniy) 

Located  in  Grand  Mesa  ABC 

2:45  pm -3:15  pm 

Break  (Tutoriai  Attendees  Oniy) 

Located  in  Atrium  Display  Area,  2“^  floor 

5:00  pm  -  6:00  pm 

Reception  (Open  to  ALL  ATTENDEES) 

Located  in  Atrium  Display  Area,  2^  floor 

1UESDAY,  NOVEMBER  17, 2009 

7:00  am  -  6:30  pm  Registration  Open 

Located  in  Grand  Mesa  Foyer,  2'"'^  floor 

7:00  am  -  8:15  am  Continentai  Breakfast 

Located  in  Atrium  Display  Area,  2^"^  floor 

8:15  am  -  8:30  am  Weicome  and  Opening  Remarks 

Located  in  Grand  Mesa  DEF,  2""^  floor 

►  Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

►  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon  Space  &  Airborne  Systems  or  Raytheon  5^5 

8:30  am  -  9:10  am  Keynote  Address 

Located  in  Grand  Mesa  DEF,  2'"'^  floor 

►  Maj  Gen  Paul  Nielsen,  USAF  (Ret),  Director,  Software  Engineering  Institute 


9:10  am  -  9:45  am  CMMi®  -  State  of  the  Modei:  The  issue  of  High  Maturity 

Located  in  Grand  Mesa  DEF,  2'"'^ floor 

►  Mr.  Clyde  Chittister,  GOO,  Software  Engineering  Institute 

►  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon  Space  &  Airborne  Systems 

9:45  am  - 1 0:1 5  am  Break 

Located  in  Atrium  Display  Area,  2^"^  floor 

1 0:1 5  am  - 1 2:00  pm  Executive  Panei 

Located  in  Grand  Mesa  DEF,  2'"'^  floor 

Moderator;  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon  Gompany 

►  Mr.  Mike  Twyman,  VP  of  the  Integrated  Gommand  and  Gontrol  Business  Unit,  Northrop  Grumman  Gorp. 

►  Mr.  David  J.  Tyler,  Sr.  Manager,  IIS  Enterprise  Process  Effectiveness,  Raytheon  Gompany 

►  Mr.  Wesley  Covell,  President  of  Defense  Programs,  Harris  Gorp. 

►  Ms.  Lynn  Penn,  Director  of  Process  Management,  Lockheed  Martin  Gorporation 

►  Mr.  Girish  Seshagiri,  GEO,  Advanced  Information  Services,  Inc. 


12:00  pm -1:30  pm  Lunch 

Located  in  Grand  Mesa  ABG 

►  Mr.  Hal  Wilson,  Director,  Engineering  Defense  Systems  Division,  Northrop  Grumman  Information  Systems 
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1  ;30  pm  -  3:00  pm 

Concurrent  Sessions 

3:00  pm  -  3:30  pm 

Break 

Located  in  Atrium  Display  Area,  2'^  floor 

3:30  pm  -  5:00  pm 

Concurrent  Sessions 

5:00  pm  -  6:30  pm 

Reception 

Located  in  Atrium  Display  Area,  2'^  floor 

WEDNESDAY,  NOVEMBER  1 8, 2009 

7:00  am  -  4:30  pm 

Registration  Open 

Located  in  Grand  Mesa  Foyer,  2"‘‘  floor 

7:00  am  -  8:00  am 

Continental  Breakfast 

Located  in  Atrium  Display  Area,  2'‘‘  floor 

8:00  am  -  9:30  am 

Concurrent  Sessions 

9:30  am  - 10:00  am 

Break 

Located  in  Atrium  Display  Area,  2'‘‘  floor 

10:00  am -11:30  am 

Concurrent  Sessions 

11:30  am -1:00  pm 

Awards  Lunch 

Located  in  Grand  Mesa  ABC 

1:00  pm  -2:30  pm 

Concurrent  Sessions 

2:30  pm  -  3:00  pm 

Break 

Located  in  Atrium  Display  Area,  2'‘‘  floor 

3:00  pm  -  4:30  pm 

Concurrent  Sessions 

4:30  pm 

Conference  adjourns  for  the  day 

THURSDAY,  NOVEMBER  19, 2009 

7:00  am  -  2:30  pm 

Registration  Open 

Located  in  Grand  Mesa  Foyer,  2"^"^  floor 

7:00  am  -  8:00  am 

Continentai  Breakfast 

Located  in  Atrium  Display  Area,  2"^"^  floor 

8:00  am  -  9:30  am 

Concurrent  Sessions 

9:30  am  -  10:00  am 

Break 

Located  in  Atrium  Display  Area,  2"^"^  floor 

10:00  am -11:30  am 

Concurrent  Sessions 

11:30  am -1:00  pm 

Lunch 

Located  in  Grand  Mesa  ABG 
►  CMMFLIVE! 

1:00  pm  -2:30  pm 

CMMi®  VI  .3  Focus  Group  Sessions 

Located  in  Grand  Mesa  F 

Members  of  the  CMMI®  user  community  will  be  afforded  a  chance  to  influence  the  content  of  the  next  release, 
VI. 3.  By  November,  the  major  elements  of  VI. 3  will  have  been  identified.  The  Focus  Group,  led  by  the  NDIA 
CMMI®  Working  Group,  and  supported  by  the  SEI,  will  provide  a  summary  of  the  major  changes  for  VI. 3. 
Conference  attendees  will  then  be  invited  to  discuss  and  provide  feedback  on  some  of  the  key  ingredients. 
Immediately  after  the  Focus  Group,  this  feedback  will  be  briefed  to  CMMI®  Steering  Group  members  to  help 
guide  prudent  change. 

2:30  pm  -  5:00  pm 

Dispiay  Dismantie 

2:30  pm 

Conference  adjourns 

MONDAY,  NOVEMBER  16, 2009 


Track  1 

Grand  Mesa  D/E 

Track  2 

Grand  Mesa  F 

Track  3 

Highlands 

Track  4 

Chasm  Creek 

Track  5 

Mesa  Verde 

Track  6 

Wind  River 

Track  7 

Wind  Star 

8:00  am  - 

1A1  -  Tutorial 

1A2  -  Tutorial 

1A3  -  Tutorial 

1A4  -  Tutorial 

1A5  -  Tutorial 

1A6  -  Tutorial 

1A7  -  Tutorial 

9:45  am 

9377  -  CMMI®,  ISO, 

9182  -  An  Agile  View 

9174  -  How  to  build 

9128 -Building 

9296  -  Unintended 

91 05 -A Step- 

9282  -  Identify 

Session  A 

Six  Sigma  and  ANSI 

of  the  CMMI®? 

and  Maintain  a 

Statistical  Support 

Consequences  of 

by-step  Tutorial 

the  Best  Leading 

748:  Soulmates  that 

Software  Center  of 

for  Organizational 

Measurement  - 

on  Planning  and 

Indicators  for  Your 

Should  to  be  Together- 

Excellence  Based  on 

Innovation 

Causes  and  Cures 

Implementing  a 

Program 

Quite  Easily! 

Seamless 

and  Deployment 

Credible  CMMI 

Integration  of  SEI 

Without  Impacting 

Appraisal 

Models  of  Excellence 

the  Innovation 

-  CMMI®,  TSP,  PSP 

‘Ereedom’ 

Mr.  Girish  Seshagiri, 

Mr.  Robert 

Advanced 

Moore,  Business 

Mr.  Tim  Kasse,  Kasse 

information  Services 

Mr.KobiVider,K.V.P 

Mr.  Riiey  Rice,  Booz 

Transformation 

Mr.  Robert 

Ms.  Nishi  Narula,  OST 

initiatives,  LLC 

inc. 

Gonsuiting 

Aiien  Hamiiton 

institute,  inc. 

Eerguson,  SEi 

10:15  am- 

1B1  -  Tutorial 

1 B2  -  Tutorial 

1 B3  -  Tutorial 

1 B4  -  Tutorial 

1 B5  -  Tutorial 

1 B6  -  Tutorial 

1 B7  -  Tutorial 

12:00  pm 

9377  -  CMMI®,  ISO, 

9182  -  An  Agile  View 

9174  -  How  to  build 

9128 -Building 

9296  -  Unintended 

91 05 -A Step- 

9282  -  Identify 

Six  Sigma  and  ANSI 

of  the  CMMI®? 

and  Maintain  a 

Statistical  Support 

Consequences  of 

by-step  Tutorial 

the  Best  Leading 

Session  B 

748:  Soulmates  that 

Software  Center  of 

for  Organizational 

Measurement  - 

on  Planning  and 

Indicators  for  Your 

Should  to  be  Together  - 

Excellence  Based  on 

Innovation 

Causes  and  Cures 

Implementing  a 

Program 

Quite  Easily! 

Seamless 

and  Deployment 

Credible  CMMI® 

Integration  of  SEI 

without  impacting 

Appraisal 

Models  of  Excellence 

the  innovation 

-CMMI®, TSP,  PSP 

‘freedom’ 

Mr.  Girish  Seshagiri, 

Mr.  Robert 

Advanced 

Moore,  Business 

Mr.  Tim  Kasse,  Kasse 

information  Services, 

Mr.KobiVider,K.V.P 

Mr.  Riiey  Rice,  Booz 

Transformation 

Mr.  Robert 

Ms.  Nishi  Narnia,  OST 

initiatives,  LLC 

inc. 

Gonsuiting 

Aiien  Hamiiton 

institute,  inc. 

Eerguson,  SEi 

1:00  pm- 

101  -  Tutorial 

1 D2  -  Tutorial 

1 C3  -  Tutorial 

1 C4  -  Tutorial 

1 C5  -  Tutorial 

1 C6  -  Tutorial 

1 C7  -  Tutorial 

2:45  pm 

9376  -The  CERT® 

9226  -Software 

9174  -  How  to  build 

9258  -  How  to 

9254  -  Multi-Model 

91 05 -A Step- 

Resiliency  Management 

Estimation  Bootcamp 

and  Maintain  a 

Achieve  Measurable 

Enhancement  of 

by-step  Tutorial 

Session  C 

Model:  Process 

Software  Center  of 

ROI  Using  Early 

Project  Management 

on  Planning  and 

Improvement  for 

Excellence  Based  on 

Defect  Detection  and 

Implementing  a 

Enterprise  Security, 

Seamless 

Defect  Prevention 

Credible  CMMI® 

Business  Continuity,  and 

Integration  of  SEI 

Appraisal 

IT  operations  to  Enable 

Models  of  Excellence 

and  Sustain  Operational 

-CMMI®, TSP,  PSP 

Resiliency 

Mr.  Girish  Seshagiri, 

Mr.  Robert 

Mr.  Wiiiiam  Deibier, 

Advanced 

Mr.  Tim  Oison,  Lean 

Moore,  Business 

Software  Systems 

information  Services, 

Soiutions  institute. 

Mr.  Tim  Kasse,  Kasse 

Transformation 

Mr.  David  White,  SEi 

Quaiity  Consuiting 

inc. 

inc. 

initiatives,  LLC 

institute,  inc. 

3:15  pm  - 

1D1  -  Tutorial 

1 D2  -  Tutorial 

1 D3  -  Tutorial 

1 D4  -  Tutorial 

1 C5  -  Tutorial 

1 D6  -  Tutorial 

1 D7  -  Tutorial 

5:00  pm 

9376  -The  CERT® 

9226  -Software 

9174  -  How  to  build 

9258  -  How  to 

9254  -  Multi-Model 

91 05 -A Step- 

Resiliency  Management 

Estimation  Bootcamp 

and  Maintain  a 

Achieve  Measurable 

Enhancement  of 

by-step  Tutorial 

Session  D 

Model:  Process 

Software  Center  of 

ROI  Using  Early 

Project  Management 

on  Planning  and 

Improvement  for 

Excellence  Based  on 

Defect  Detection  and 

Implementing  a 

Enterprise  Security, 

Seamless 

Defect  Prevention 

Credible  CMMI 

Business  Continuity,  and 

Integration  of  SEI 

Appraisal 

IT  operations  to  Enable 

Models  of  Excellence 

and  Sustain  Operational 

-CMMI®, TSP,  PSP 

Resiliency 

Mr.  Girish  Seshagiri, 

Mr.  Robert 

Mr.  Wiiiiam  Deibier, 

Advanced 

Moore,  Business 

Software  Systems 

information  Services, 

Mr.  Tim  Oison,  Lean 

Mr.  Tim  Kasse,  Kasse 

Transformation 

Mr.  David  White,  SEi 

Quaiity  Consuiting 

inc. 

Soiutions  institute,  inc. 

initiatives,  LLC 

institute,  inc. 

TUESDAY,  NOVEMBER  1 7, 2009 


Track  1 

Track  2 

Track  3 

Track  4 

Track  5 

Track  6 

Track  7 

Grand  Mesa  D/E 

Grand  Mesa  F 

Highlands 

Chasm  Creek 

Mesa  Verde 

Wind  River 

Wind  Star 

CMMI®  and 
Process 
Improvement 

Praetieal  Guidanee 

GMMI® 
Economics  & 
Business  Value 

High  Maturity 

GMMI® 
Government  & 
Aequisition 

Appraisals 

GMMI  VI  .3  Topic 
&  GMMI-SVGs, 
Lean 

Session  Chair: 

Mr.  Jack  Ferguson, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Gene  Miluk, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Geoff  Draper, 
Flarris  Corporation  & 
Mr.  Bob  Ferguson, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Dennis 

Goldenson,  Software 
Engineering  Institute 
&  Mr.  Fred  Schenker, 
Software  Engineering 
Institute 

Session  Chair: 

Ms.  Lorraine 
Adams,  Software 
Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software 
Engineering 
Institute  &  Mr.  Ken 
Weinberg,  Raytheon 
Corporation 

Session  Chair: 

Ms.  Susan 
Bassham,  U.S. 

Army  Aviation  & 
Missile  Command 

1:30  pm  - 
2:15  pm 

9386  -  CMMI® 
for  Large-Scale/ 
Systems  of  Systems 
Engineering  Projects 

91 79 -Work  On  Your 
Engineering  Business, 
Not  In  It 

9147  -  Dynamic 

Program  Schedule, 
Costand  Returns 
Analysis 

9146- Goal - 
Question  -  Model 

8806  -  Benefits  to 
the  Evolution  of  Fligh 
Maturity  Software 
Development:  A 15 

Year  Case  Study 

9398  -  Hocus  Pocus, 
What’s  With  All  The 
Issues  About  Non- 
Focus? 

CMMI  VI  .3-  From 
the  Past  to  the  Future 

Mr.  Patrick  McCusker, 
Booz  Allen  Hamilton 

Mr.  Rolf  Re itzig, 
Cognence,  Inc. 

Mr.  Philip  Fahringer, 
Lockheed  Martin 
Corporation 

Mr.  Michael  Campo, 
Raytheon  Company 

Mr.  Daniel  Drew, 

United  Space  Alliance 

Mr.  Paul  Byrnes, 
Integrated  Systems 
Diagnostics,  Inc. 

Mr.  Mike  Phillips, 
Software  Engineering 
Institute 

2:15  pm- 
3:00  pm 

Making  the  CMMI® 

Sing  -  A  Framework  for 
Performance  Excellence 
Parti  (Part  2  is  to  be 
given  on  Tuesday  - 
Track!  at  3:30 -4:1 5 
p.m.) 

Mr  Jeff  Dutton, 

Jacobs  Technology  Inc. 

Process-Performance 
Based  Reliability 

Mr.  William  Winkel, 
Northrop  Grumman 
Corporation 

91 84 -The 

Economics  of 

CMMI® 

Mr.  Mike  Campo, 
Raytheon  Company 

9389  -  Making 

CMMI®  Level  5 
Statistical  Principles 
Palatable  to  an 
Employee-Wide 
Demographic 

Ms.  Deepti  Sharma, 
OST 

9306  -  Directive 
Documents  and  ITAR 
Made  Easy 

Mr.  Kenneth 

Weinberg,  Raytheon 

9136  -  Making  the 
Most  of  GP3.2 

Ms.  Susan  Byrnes 
Natural  SPI,  Inc. 

91 67  -  Lessons 
Learned  Piloting  the 
CMMI®  for  Services 

Ms.  Diane  Mizukami 
(Williams),  Northrop 
Grumman  Information 
Systems 

CMMI®  and 
Proeess 
Improvement 

Praotioal  Guidanoe 

GMMI® 
Eeenomies  & 
Business  Value 

High  Maturity 

GMMI® 
Government  & 
Aequisition 

Appraisals 

GMM®-SVGs, 
Lean,  SmSettings, 
etc. 

Session  Chair: 

Mr.  Jack  Ferguson, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Gene 

Miluk,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Geoff  Draper, 
Flarris  Corporation 
&  Mr.  Bob 
Ferguson,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Dennis 

Goldenson,  Software 
Engineering 
Institute  &  Mr.  Fred 
Schenker,  Software 
Engineering  Institute 

Session  Chair: 

Ms.  Lorraine 
Adams,  Software 
Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software 
Engineering 
Institute  &  Mr.  Ken 
Weinberg,  Raytheon 
Corporation 

Session  Chair: 

Ms.  Susan 
Bassham,  U.S. 
Army  Aviation  & 
Missile  Command 

3:30  pm  - 
4:15  pm 

Making  the  CMMI® 

Sing  -  A  Framework 
for  Performance  Excel¬ 
lence  Part  2  (Part  1  was 
given  on  Tuesday  -  Track 

1  at  2:1 5-  3:30p.m.) 

Mr.  Jeff  Dutton, 

Jacobs  Technology,  Inc. 

9391  -  Choices  to 
be  Made  in  CMMI® 
Adoption 

Dr.  Rick  Hefner, 
Northrop  Grumman 
Corporation 

9185 -CMMI® 
Economics  101: 
CMMI®  for 

Executives 

Mr.  Geoff  Draper, 

Harris  Corporation 

9116  -  Changing 
Behavior:  The  Key  to 
Adoption  Complex 
Process  Technology 

Dr.  Gene  Miluk,  SEI 

8907  -  How  CMMI® 
was  Used  for  Process 
Improvement  In 
the  Support  of 
Government-Wide 
Acquisition  Contract 
(GWAC)  Vehicles 

Mrs.  Sharon  Cobb 
Flanagan,  SAIC 

9130  -  Supporting 
the  High  Maturity 
Process  Improvement 
and  Understanding 
the  Application 
SCAMPISM  Method 

To  It 

Mr.KobiVidar,K.V.P. 

Consulting 

9126 -CMMI® 
for  Services:  An 
Approach  to  Improve 
Your  Program 
Management  Qffice 

Ms.  Patricia  Mitryk, 
Cognence,  Inc. 

4:15  pm  - 
5:00  pm 

931 2 -CMMI®  in  a 
Small  Company:  The 
Cobbler’s  Children  Can 
Flave  Shoes  (And  Best 
Practices) 

Mr.  Michael  Knox, 
TECHSOFT,  Inc. 

9298  -  Assurance  for 
CMMI®:  A  Toolbox 
for  Multiple  Cyber 
Challenges 

Mrs.  Michele  Moss, 
Booz  Allen  Hamilton 

9401  -  Achieving 

Quality  QPPQ  via 
Effective  Usage  of 

PPBs  and  PPMs 

Dr.  Bin  Cong,  CBS 

Tech 

9403  -  Tailoring 
CMMI®  for  an 
Enterprise  Resource 
Planning  COTS 
Software  Environment 

Ms.  Alison  L.  Schwier, 
US.  Army 

9385  -  Reducing  the 
Cost  and  Increasing 
the  Value  of  CMMI 
Re-Appraisals 

Mrs.  Beth  Layman, 
Layman  &  Layman 

9299  -  Creatively 
Applying  CMMI®- 
SVC  in  a  Very  Small 
Consulting  Firm 

Mr.  Bill  Smith, 

Leading  Edge  Process 
Consultants 
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Pligh  Maturity 

GMMI® 
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Acquisition 
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GMMI®  -  SVGs, 
Lean,  SmSettings, 
etc. 

Session  Chair: 

Mr.  Jack  Ferguson, 
Software 

Engineering  Institute 

Session  Chair: 

Mr.  Gene 

Miluk,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Geoff  Draper, 
Harris  Corporation 
&  Mr.  Bob 
Ferguson,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Dennis 

Goldenson,  Software 
Engineering 
Institute  &  Mr.  Fred 
Schenker,  Software 
Engineering  Institute 

Session  Chair: 

Ms.  Lorraine 
Adams,  Software 
Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software 
Engineering 
Institute  &  Mr.  Ken 
Weinberg,  Raytheon 
Corporation 

Session  Chair: 

Ms.  Susan 
Bassham,  U.S. 
Army  Aviation  & 
Missile  Command 

8:00  am  - 
8:45  am 

9304  -  Sustainment 
and  Continued 
Institutionalization  of 
Best  Practices  and 
CMMI®atSPAWAR 

9138  -  Fligh  Velocity 

Performance 

Improvement 

9213  -  QPMingYour 
SEPG 

9217  -  A  Taxonomy  of 
CMMI®  High  Maturity 
Performance  Models 

0000  -  Using  CMMI® 
for  Acquisition 
in  Integration 
Organizations 

9371  -  Appraisals 
and  CMMI®  Gotchas 
-  Lessons  in  CMMI® 
Use  and  Appraisal 
Preparation 

9396  -  Applying  the 
CMMI®  for  Services 
to  the  Process  Group 
(Physician,  Heal 
Thyself!) 

Mr.  Michael  Kutch, 
Space  &  Naval 

Warfare  Systems 
Center  -  Atlantic 

Mr.  Jeff  Dutton 

Jacobs  Technology 

Inc. 

Mr.  Craig  Hale, 

Esterline  Control 
Systems  -AVISTA 

Dr.  Richard  Welch, 
Northrop  Grumman 
Corporation 

Mr.  Steve  Kelley, 
Northrop  Grumman 
Corporation 

Mr.  Neil  Potter,  The 
Process  Group 

Dr.  Rick  Hefner, 
Northrop  Grumman 
Corporation 

8:45  am  - 
9:30  am 

9178 -CMMI® 
Process  Improvement: 
Its  not  a  technical 
Problem,  It’s  a  People 
Problem! 

9202  -  Statistical 
Tune-Up  of  the  Peer 
Review  Process 

9223  -  We’re  Already 
There:  Matching 
Existing  High  Maturity 
Behaviors  to  the 
CMMI®  Model 

9245  -  The  Selection 
and  Deployment  of  a 
Standard  COTS  Monte 
Carlo  Software  Tool 

8741  -  Leveraging 
CMMI®  for 

Acquisition  to  Improve 

Organizational 

Workforce 

Performance 

9311  -  Piloting 
Results-Based 
Appraisals 

9274  -  An  Overview 
of  CMMI®-SVC 
for  CMMI®-DEV 
Enthusiasts 

Mr  Rolf  Re itzig, 
Cognence,  Inc. 

Mr.  Tom  Lienhard, 
Raytheon  Missile 
Systems 

Mr.  Bradley  Bittorf 
Raytheon  Company 

Mr.  Ered  Oleson,  BAE 
Systems,  Inc. 

Dr.  Kenneth  Nidiffer, 

SEI 

Mr.  Larry  McCarthy 
Global  Software 

Group 

Mr.  Bill  Smith, 

Leading  Edge  Process 
Consultants 

CMMI®  and 
Proeess 
Improvement 

Practical  Guidance 

GMMI® 
Eeenemies  & 
Business  Value 

High  Maturity 

GMMI® 
Government  & 
Acquisition 

Appraisals 

GMMI®  -  SVGs, 
Lean,  SmSettings, 
etc. 

Session  Chair: 

Mr.  Jack  Ferguson, 
Software 

Engineering  Institute 

Session  Chair: 

Mr.  Gene 

Miluk,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Geoff  Draper, 
Harris  Corporation 
&  Mr.  Bob 
Ferguson,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Dennis 

Goldenson,  Software 
Engineering 
Institute  &  Mr.  Fred 
Schenker,  Software 
Engineering  Institute 

Session  Chair: 

Ms.  Lorraine 
Adams,  Software 
Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software 
Engineering 
Institute  &  Mr.  Ken 
Weinberg,  Raytheon 
Corporation 

Session  Chair: 

Ms.  Susan 
Bassham,  U.S. 
Army  Aviation  & 
Missile  Command 

10:00  am- 
10:45  am 

9106  -  The  Uses 
of  the  Peer  Review 
beyond  CMMI® 

Mr.  Paul  Nugent 
General  Dynamics 
Corportation 

9275  -  CMMI®  in 
the  Social  Media  (For 
the  Social  Media- 
Challenged!) 

Mr.  Bill  Smith, 

Leading  Edge 

Process  Consultants 

9190 -CMMI® 
Economics  501 :  High 
Maturity 

Mr.  Mike  Campo, 
Raytheon  Company 

9232  -  Piloting  a 

Hybrid  Requirements 
Engineering  Process 
for  Translating 
Qualitativem 

Information  into 
Quantitative 

Performance  Measures 

Mr.  Dennis  Goldenson, 
SEI 

9301  -  Improving 
Processes  the  NSA 

Way 

Mr.  Robert 

Moore,  Business 
Transformation 

Institute,  Inc. 

9129  -  Using  the 
SCAMPI  Method  to 
lead  CMMI®  Multi- 
Constellations  with 
Additional  Standards 
Progress  Check  and 
Appraisal 

Mr.KobiVidar,K.V.P. 

Consulting 

9137-ITILV3.0 
Compliance 
Benchmarking  with 
CMMI-SVC  SCAMPI  A 

Mr.  Jeff  Dutton, 

Jacobs  Technology, 

Inc. 

10:45  am- 
11:30  am 

9246  -  Integrating 
Corporate  Goals  and 
Processes  Using  the 
Engineering  Lifecycle 
Vee  Model 

Dr.  Kevin  Forsberg, 

The  Center  for 

Systems  Management 

9214 -Hi,  my 
name  is  Root  Cause 
Analysis.  Have  we 
met? 

Mr.  Craig  Hale, 

Esterline  Control 
Systems  -AVISTA 

9378  -  Using 

Corporate  Finance 
Principles  to  Easily 
Determine  Return  on 
Investment  (ROI) 

Ms.  Deepti  Sharma, 
OST 

91 68  -  How  1  Created 
Our  Peer  Review 
Baselines  and  Models 

Ms.  Diane  Mizukami 
(Williams),  Northrop 
Grumman  Information 
Systems 

9114  -  Implementing 
Requirements 
Management  To  Deliver 
Life  Cycle  Software 
Solutions  That  Ensure 
Warfighting  Superiority 
and  Information 
Dominance:  How  We 
Moved  The  Rock 

Mr.  Harlan  Black,  U.S. 
Army  CECOM 

9234 -A View 
from  the  Trenches: 
Practical  Guidance 
for  Appraisal  Artifact 
Management 

Mr.  David  Dickinson, 
Northrop  Grummon 
Corporation 

9397  -  Strategies 
for  Transitioning  to 
CMMI-SVC 

Dr.  Rick  Hefner, 
Northrop  Grumman 
Corporation 

Track  1 

Grand  Mesa  D/E 

Track  2 

Grand  Mesa  F 

Track  3 

Highlands 

Track  4 

Chasm  Creek 

Track  5 

Mesa  Verde 

Track  6 

Wind  River 

Track  7 

Wind  Star 

CMMI®  and 

Practical  Guidance 

CMMI® 

Pligh  Maturity 

CMMI® 
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CMMI®  -  SVCs, 

Process 

Eccncmics  & 

Government  & 

Lean,  SmSettings, 

Improvement 
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Aequisition 

etc. 

Session  Chair: 

Session  Chair: 

Session  Chair: 

Session  Chair: 

Mr.  Dennis 

Ms.  Lorraine 

Mr.  Ken  Nidiffer, 

Mr.  Geoff  Draper, 

Goldenson,  Software 

Adams,  Software 

Software 

Session  Chair: 

Session  Chair: 

Session  Chair: 

Harris  Corporation 

Engineering 

Engineering 

Engineering 

Ms.  Susan 

Mr.  Jack  Ferguson, 

Mr.  Gene 

&  Mr.  Bob 

Institute  &  Mr.  Fred 

Institute  &  Mr.  Mike 

Institute  &  Mr.  Ken 

Bassham,  U.S. 

Software 

Miluk,  Software 

Ferguson,  Software 

Schenker,  Software 

Phillips,  Software 

Weinberg,  Raytheon 

Army  Aviation  & 

Engineering  Institute 

Engineering  Institute 

Engineering  Institute 

Engineering  Institute 

Engineering  Institute 

Corporation 

Missile  Command 

1:00  pm- 

9379  -  NAVAIR’s 

8787  -  Improving 

8909  -  Consistency  in 

921 6 -ABCs  Of 

8764  -  Predicting 

9287  -  Flow  to 

9196  -  Interpretation 

1:45  pm 

Process  Asset  Library 

Process 

Quality  Assessments 

Process  Performance 

Quality 

Prepare  for  a  CMMI® 

and  Lesson  Learned 

(PAL),  A  Step  Toward  A 

Institutionalization 

Models 

SCAMPI  A:  Applying 

from  Fligh  Maturity 

Corporate 

Through  Process 

Agile  Concepts  to 

Implementation  of 

Organizational  Set  of 

Training 

Save  Time  and  Money 

CMMI-SVC 

Standard  Processes 

(OSSP) 

Dr.  Richard  Welch, 

Mr.  Donald  Beckett, 

Ms.  Michele  Shaw. 

Ms.  Judy  Overhauser- 

Ms.  Ellen  Chilikas, 

Mrs.  Debra  Perry, 

Northrop  Grumman 

Quantitative  Software 

Eraunhofer  Center, 

Mr.  Kobi Picker,  K.V.P. 

Duett,  NAVAIR 

Raytheon  Company 

Harris  Corporation 

Corporation 

Mgt. 

Maryland 

Consulting 

1:45  pm  - 

9153 -After  13 

9291  -  “You  Say 

8871  -  MSI 

9294  -  Using  Flidden 

9326  -  “Patience  is  a 

9365  -  Streamlining 

9264  -  Applying 

2:30  pm 

years,  I  have 

Tomato,  1  Say  Eggplant: 

Execution:  Change 

Markov  Models  as 

Virtue”-  A  Day  in  the 

Processes  and 

Lean  Principles  to  the 

learned...  Tools 

Comparing  Process 

Flappens,  Flow  to  Deal 

a  Statistical  Process 

Life  of  a  SEPG  Lead 

Appraisals 

CMMI®  for  Services 

do  not  solve  your 

References  for 

with  It 

Control  Technique: 

and  ITIL 

problems 

Systems  Engineers  and 

An  Example  from  a  ML 

Project  Managers  in 

5  Organization 

a  CMMI®-  Compliant 

Mr.  John  Bethmann, 

Organization” 

Mr.  Robert 

Concurrent 

Moore,  Business 

Ms.  Tammye 

Mr.  Tim  Olson,  Lean 

Technologies 

Mr.  Peter  Henry  BAE 

Ms.  Jill  Brooks, 

Transformation 

Thornton,  Department 

Mr.  Gary  Natwick, 

Solutions  Institute, 

Corporation 

Systems 

Raytheon  Company 

Institute,  Inc. 

of  Navy 

Harris  Corporation 

Inc. 

CMMI®  and 

Practical  Guidance 

CMMI® 

Pligh  Maturity 

CMMI® 

Appraisals 

Multi-Mcdels 

Process 

Eccncmics  & 

Government  & 

Improvement 

Business  Vaiue 

Aequisition 

Session  Chair: 

Session  Chair: 

Session  Chair: 

Session  Chair: 

Mr.  Dennis 

Ms.  Lorraine 

Mr.  Ken  Nidiffer, 

Mr.  Geoff  Draper, 

Goldenson,  Software 

Adams,  Software 

Software 

Session  Chair: 

Session  Chair: 

Session  Chair: 

Harris  Corporation 

Engineering 

Engineering 

Engineering 

Ms.  Susan 

Mr.  Richard 

Mr.  Gene 

&  Mr.  Bob 

Institute  &  Mr.  Fred 

Institute  &  Mr.  Mike 

Institute  &  Mr.  Ken 

Bassham,  U.S. 

Barbour,  Software 

Miluk,  Software 

Ferguson,  Software 

Schenker,  Software 

Phillips,  Software 

Weinberg,  Raytheon 

Army  Aviation  & 

Engineering  Institute 

Engineering  Institute 

Engineering  Institute 

Engineering  Institute 

Engineering  Institute 

Corporation 

Missile  Command 

3:00  pm  - 

9387  -  10.5  Process 

9354  -  Strategies  for 

91 77  -  Lessons 

9407  -  Multi-Attribute 

9321  -  Enterprise 

9189 -CMMI® 

9203  -  Flow  Rocket 

3:45  pm 

Improvement 

Process  Definition 

Learned  Using  Earned 

Modeling  and 

Architecting  and  the 

Economics  202: 

Scientist  Implement 

Mistakes  from  Top 

and  Deployment  Part 

Value  Management 

Practical  Use 

Incorporation  of  Early 

Appraisals 

Fligh  Maturity 

Executives? 

1  (Part  1  is  to  be  given 

System  to  Track  Effort 

Systems  Engineering 

on  Wednesday  -  Track  2 

and  Schedule  Weekly 

Data  Into  the 

at  3:45  -  4:30  p.m. ) 

at  the  Individual  and 

Leadership  Decision 

Team  Level  and  Be 

Making  Process 

Able  to  Detect  a  One- 

During  Concept 

Day  Schedule  Slip 

Development 

Mr.  Girish  Seshagiri, 

Mr.  David  Sobetski, 

Mr.  George  Ereeman, 

Mr.  Tom  Lienhard, 

Mr.  Carlos  Caram, 

Mr.  Ered  Schenker, 

Advanced  Information 

General  Dynamics 

United  States  Air 

Mr.  Geoff  Draper, 

Raytheon  Missile 

CSD  BRASIL 

SEI 

Services  Inc. 

Land  Systems 

Eorce 

Harris  Corporation 

Systems 

3:45  pm  - 

9144  -  Transitioning 

Strategies  for  Process 

9188 -CMMI 

9390  -  Discrete  Event 

9154  -  Simplifying 

9227  -The  ABC’s 

4:30  pm 

From  a  CMMI® 

Definition  and 

Economics  203: 

Simulation  for  QPM 

Lifecycle  Definition 

of  Class  C,  Class  B, 

Implementerto  an 

Deployment  Part  2 

Model  Tailoring 

-  Can  it  really  be  that 

Process 

Class  A’s:  Metrics 

Appraiser 

(Part  1  was  given  on 

easy? 

and  Lessons  Learned 

Wednesday  -  Track  2  at 

from  Appraisal  Event 

3:00  -  3:45  p.m.) 

Scheduling 

Mr.  Warren  Scheinin, 

Mr.  John  Bethmann; 

Northrop  Grumman 

Mr.  Ered  Schenker, 

Mr.  Jeff  Dutton,  Jacob 

Ms.  Deepti  Sharma, 

Concurrent 

Mr.  Bradley  Bittorf, 

Corporation 

SEI 

Technologies,  Inc. 

OST 

Technologies ,  Corp 

Raytheon  Company 
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Session  Chair: 

Mr.  Jack  Ferguson, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Gene  Miluk, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Geoff  Draper, 
Harris  Corporation  & 
Mr.  Bob  Ferguson, 
Software  Engineering 
Institute 

Session  Chair: 

Mr.  Dennis 

Goldenson,  Software 
Engineering  Institute 
&  Mr.  Fred  Schenker, 
Software  Engineering 
Institute 

Session  Chair: 

Ms.  Lorraine 
Adams,  Software 
Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software 
Engineering 
Institute  &  Mr.  Ken 
Weinberg,  Raytheon 
Corporation 

Session  Chair: 

Ms.  Susan 
Bassham,  U.S. 
Army  Aviation  & 
Missile  Command 

8:00  am  - 
8:45  am 

9313  -  Mission 
Readiness  Through 
Integrated  Decision 
Making  ...  BIM,  BAM 
and  ROW! 

Mr.  James  Watson, 
Facility  Lifecycle 

Group  Div.  of 
MACTECE&C 

9308-  Post  Merger 
Process  Syndrome: 
Integreating 
&  Refining 
Organizational 
Processes 

Mr.  Michael  Kutch, 
Space  &  Naval 

Warfare  Systems 
Genter  -  Atlantic 

9181  -  Are  You  Doing 
R&D,  or  Catch-up 
&  D?  Are  you 

Building  Software,  or 
Flopeware? 

Mr.  Rolf  Re itzig, 
Cognence,  Inc. 

9143  -  Using  Moving 
Average  Models 
to  Predict  Process 
Performance 

Mr.  Robert  Tuthill, 
Northrop  Grumman 
Corporation 

9359  -  Moving  your 
Security,  Business 
Continuity,  and  IT 
Activities  to  the 

Next  Level  with  the 
CERT®  Resiliency 
Management  Model 

Ms.  Gibbie  Lu  Hart, 

SEI 

9373  -  Lockheed 

Martin  Aeronautics 
Appraisal  Project 
Management  Strategy 

Ms.  Pam  Hudson, 
Lockheed  Martin 
Aeronautics 

9257  -  The  Next  Step 
in  Process  Evolution: 
CMMI®  andTSP/PSP 

Mr.  Jeffrey  Schwalb, 
Naval  Air  Systems 
Command 

8:45  am  - 
9:30  am 

91 65 -Make  PIIDs 

Easy  -  No  Surprises! 

9141  -  Tools  and 
Implementation 
Strategies  for 

Process  Improvement 
via  CMMI®  for 
Comprehensive 
Software  Lifecycle 
Management 

9283  -  CMMI® 
Measurement  and 
Metrics 

9148 -Use  of  Monte 
Carlo  Simulation  for  a 
Peer  Review  Process 
Performance  Model 

9366  -  Enjoy  the 
Scenery  on  the  Path 
to  High  Maturity 

9369  -  Lockheed 

Martin  Aero  Standard 
Approach -A 

Strategy  to  Select 
Cbjective  Evidence  for 
the  PHD 

9204  -  Everything 

You  Wanted  to  Know 
About  CMMI®  and 

Six  Sigma  but  Did  Not 
Know  Who  to  Ask 

Ms.  Vicki  Gaianko, 
Lockheed  Martin, 
IS&GS-Givil 

Mrs.  Denise  Padilla, 
Sandia  National 
Laboratories 

Dr.  Elliot  Lynn, 
GEGOMSEG 

Ms.  Emerald  Russo, 
BAE  Systems 

Ms.  Susan  Bassham, 
U.S.  Army  Aviation 
and  Missile  Command 

Mrs.  Pam  Hudson, 
Lockheed  Martin 
Aeronautics 

Mr.  Tom  Lienhard, 
Raytheon  Missile 
Systems 

9:30  AM  - 10:00  AM  BREAK  IN  ATRIUM  DISPLAY  AREA 

10:00  am- 
10:45  am 

9346  -  Exploiting 
Decision  to 
Requirements 
Traceability 

9324  -  Measuring 

True  Agility  in 

Agile  Software 
Development 

9163  -  Picking 
the  Right  Process 
Improvements 

9211  -  Transforming 
Your  Way  to  Control 
Charts  that  Work 

91 87 -Level  5 
the  Hard  Way  - 
Persevering  through 
Crganizational 

Changes 

9266  -  Rapidly 
Implementing  Lean 
CMMI®  Processes 

That  Meet  Business 
Needs 

Mr.  John  Fitch,  SAIG 

Mr.  Robert 

Moore,  Business 
Transformation 
Institute,  Inc. 

Mr.  Joseph  Vandeville, 
Northrop  Grumman 
Corporation 

Mr.  Richard  Welch, 
Northrop  Grumman 
Corporation 

Ms.  Dorna  Witkowski, 
Lockheed  Martin 
Corporation 

Mr.  Tim  Olson,  Lean 
Solutions  Institute, 

Inc. 

10:45  am- 
11:30  am 

Strategies  for  Process 
Definition  and 
Deployment  Part  2 
(Part  1  was  given  on 
Wednesday  -  Track  7  at 
3:45  -  4:30  p.m.) 

9244  -  Perspectives 
on  Use  and 
Crganizational  Impact 
of  Measurement  and 
Analytical  Methods 
in  CMMI®  Fligh 

Maturity  Crganizations 

9292  -  Systems 
Engineering 

Processes 

Improvement  using 
the  CMMI®  in  large 
System 

of  Systems  Space 
Programs 

9383  -  What?  1 

Need  to  Create  an 
Appraisal  Database 
Containing  Thousands 
of  Artifacts!  HELP!... 
Sensible  PHD 

Strategies 

9394  -  Comparing 
Scrum  and  CMMI® 

-  How  Can  They  Work 
Together 

Mr.  Alfred  Schenker, 

SEI 

Dr.  Dennis  Goldenson, 
SEI 

Ms.  Revital  Goldberg, 
Israel  Aerospace 
Industries 
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Lean  Solutions  Institute,  Inc.  (LSI)  specializes 
in  helping  organizations  to  rapidly  achieve 
measurable  results  by  using  benchmarking 
and  Lean  Solutions™  (e.g.,  best  practices  to 
implement  CMMI®  in  a  lean  way)  to  successfully 
improve  client  products  and  services.  LSI  helps 
organizations  to  measurably: 


•  Achieve  ROI  (e.g.,  7:1) 

•  Increase  productivity,  performance  and  quality 

•  Reduce  cycle  time/schedule 

•  Reduce  defects  (e.g.,  post-release  defects),  rework  and  costs  of  poor  quality 

•  Achieve  world-class  results  (e.g.,  70-90%  defect  removal  efficiency  or  defects  removed  before  test) 


Systems  engineering  and  software  engineering  have  become  more  and  more  complex  over  the  years.  With  this  growing 
complexity,  processes  and  procedures  have  become  larger  and  more  complex.  Based  on  surveys,  most  organizations  do  not 
like  their  processes  and  procedures  (e.g.,  including  CMMI®  Maturity  Level  3-5  organizations)  and  they  can  have  some  of  the 
following  lean  problems: 

•  Too  large  and  complex  (i.e.,  not  lean  or  agile) 

•  Have  non- value  added  activities 

•  Lack  of  visualization  (e.g.,  pictures,  diagrams,  tables,  charts,  etc.) 

•  Difficult  to  use  (e.g.,  poor  usability) 

•  Lack  of  “chunking”  which  is  a  best  practice  for  usability  (7  plus  or  minus  2  principle) 

•  Lack  of  innovation 

•  Lack  of  “good  metrics”,  not  the  right  metrics,  or  not  lean  metrics 

LSI  has  a  patent  pending  approach  for  defining  systems  engineering  and  software  engineering  processes  (e.g.,  CMMI® 
compliant  processes)  in  a  lean  (e.g.,  short,  usable,  visual)  way.  Although  this  approach  can  be  simple,  it  also  scales  up  to  handle 
complex  processes  (e.g.,  NASA  processes).  LSI  uses  “good  diagrams”  (i.e.,  process  models)  for  putting  the  5  W’s  (who,  what, 
where,  when,  why)  on  one  page.  These  visual  one-page  diagrams  along  with  a  page  of  support  text  typically  replace  about  25- 
30  pages  of  text.  For  example,  lean  CMMI®  processes  typically: 

•  Cost  33%-50%  of  a  typical  CMMF  implementation 

•  Take  half  the  time  to  implement  (e.g.,  1  year  instead  of  2  years) 

•  Are  20-25%  of  the  size  of  a  typical  CMMF  implementation 

In  several  CMMI®  success  stories  (independently  verified)  using  the  LSI  approach,  organizations  estimate  that  processes  are 
about  20-25%  of  the  size  of  sister  business  units  with  a  similar  CMMI®  rated  processes,  and  have  achieved  CMMI®  maturity 
levels  in  half  the  time  (or  less). 


LSI  can  help  your  organization  achieve  measurable  results,  reduce  size  and  complexity,  and  improve  processes  and  metrics  to 
become  much  more  lean,  “value  added”,  visual,  and  usable.  LSI  also  uses  an  ISO/Baldrige  approach  to  implementing  CMMI®, 
LSI  only  does  improvement  and  uses  independent  Authorized  SEI  Lead  Appraisers  to  objectively  verify  LSI  Lean  Solutions™ 
for  CMMI®. 


Lean  Solutions  Institute,  Inc.  (LSI) 
(760)  804-1405 
www.LSI-INC.com 
Info@LSI-INC.com 


CMMI®  is  a  registered  trademark  of  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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SEI  Partner  Network 


Project  Engineering  Metrics  (PEM) 


Engineering  Process  Group  (EPG) 


^Software  Engineering  Process  Group  (SEPG) 


Process  Organization 
Process  Tools  &  Media 
Process  Advancement 
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CMMI®  Benefits  -  industry 


HARRIS 


Proven  framework  for  program  planning  and  execution 

-  Provides  infrastructure  for  best  practices 

-  Helps  evaluate  and  set  direction  for  process  improvement 
Performance-focused 

-  Measurable  impacts  on  business  effectiveness 

-  Cost,  schedule,  quality,  predictability 
Levels  the  playing  field 

-  Brings  integrity  and  discipline  to 
competition  and  program  execution 

Common  language  for  communicating  ~ 

across  company  boundaries 

-  Facilitates  teaming,  process  integration,  and  benchmarking 


CMMI, 


SEI  Partner 
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CMMI®  Benefits  -  Harris 


HARRIS 


•  Integration  of  cross-functional  business  processes 

-Alignment  and  flow-down  of  objectives 

-All  functions  engaged,  with  annual  improvement  plans  regularly 
monitored 

-Improved  communication  paths  and  stakeholder  involvement 

•  Identification  of  process  gaps  and  institutionalization  needs 

-Plan  the  plan;  validation  vs.  verification;  peer  reviews;  defect 
analysis 

•  Self-governance 

-Accountability  for  process  compliance,  risks,  organizational 
performance 

-Greater  management  visibility  and  oversight  of  program  execution 

•  Continuous  process  improvement 

-Productivity,  quality,  predictability 

-Leading  indicators  of  performance 
issues  (fewer  surprises) 


Process  Matters 

Division  Process  Group  (DPG) 


rJ>  Exi 
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Addressing  Program  Performance  HAfU^S 


Program  Start-Up  Support 


p^Qp|_^  *  Customer  Intimacy 

>  Integrated  Program  Teams 
■Skills  Assessment  /  Training 

>  Integrated  Talent  Management 


Cross-Functional 

on  me  ground  i¥i^i  i  LWi  1 1 

'Making  TrAcJu- 

Process  Integration 

/  X 

Media  Accessibility 

GCSO  INTEGRATED 
PROCESS  MANUAL 


PROCESS 


Program  Readiness 
Level 

1^3 

■ 

1 
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CMMI,  Lean  Six  Sigma 
Command  Media 
Risk  Management 
Reviews 
Validation 


Process  Compliance 
Monitor 


Technology  Readiness 
Level 


HJ 


TECHNOLOGY 

•  Centers  of  Excellence 

•  Standard  Tools 

•  Technology  Readiness 

•  Program  Readiness 

•  IR&D 
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Process  &  Program  Performance 

NDIA  SE  Effectiveness  Study  (Harris  Projects^ 


HARRIS 


Correlations  of  SE  Effectiveness  with  Program  Performance 


i_R^qts_+_Tech  Sdution 
Technical  Solution  (SEC-TS) 
Requirements  Dev  &  Mgmt  (SEC-REQ) 
Product  Architecture  (SEC-ARCH) 
^^Rej^^jjii^cFSolutio^ 
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Product  Integration  (SEC-PI) 
r  OveralfS^' 
.Configuration  Management  (SEC-CM) 


Verification  (SEC-VER) 

Validation  (SEC-VAL) 

Project  Planning  (SEC-PP) 
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SEC-CMMI 
Risk  Management  (SEC-RSKM) 

SEC-EXP 
IPT  Capability  (SEC-IPT) 
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Project  Monitoring  &  Control  (SEC-PMC)  -Q  1 1 9  e 

-0.200 


Strong 
Correlation 
with  Harris 
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Correlation 
with  Harris 
Program 
Performance 


0.451 
0.482^ 


Front-end  processes 
have  highest 
correlation  with  Harris 
program  performance 
(left  side  of  “vee”) 


'0.402 

'0.399 


0.000 


0.200 


0.400 


0.600 


0.800 
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Composite  Measures 


□  Correlation  (r)  ■  Chance  Probability  (p) 


Reference:  A  Survey  of  Systems  Engineering  Effectiveness,  Initial  Results.  Software  Engineering  Institute,  CMU/SEI-2008-SR-034. 
http://www.sei.cmu.edu/librarv/abstracts/reports/Q8srQ34.cfm 
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Program  Readiness  Assessment  HAt^RtS 
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27-Oct-09 


Summary 


HARRIS 


*  It’s  a  journey  -  not  there  yet 

-  Committed  to  continuous  improvement 

-  Executive  involvement  has  grown 

*  Too  heavy  for  broad  application  across  fuii  range  of 
program  types 

-  $2M  to  billions 

-  3-month  quick  reacts,  Agile  development,  large  scale  HW/SW 
development 

*  Striving  to  find  proper  baiance  of  process,  toois,  metrics 

-  Enough  to  be  compliant  and  efficient  and  ensure  successful 
program  execution 

-  Cost-effective  appraisals  and  compliance  oversight 


CMMI  &  process  focus  are  a  necessary  but  not  wholly 
sufficient  condition  for  successfui  program  execution 
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Five  Operating  Sectors 


Large  Scale  Systems 
Integration 

C^ISR 

Unmanned  Systems 

Airborne  Ground 
Surveillance  /  C2 


Radar  Systems 
C^ISR 

Electronic  Warfare 
Naval  &  Marine  Systems 


Command  &  Control 
Systems 

Network  Communications 

Intelligence,  Surveillance  & 
Reconnaissance  Systems 


Naval  BMC2  Navigation  &  Guidance 

Global  /  Theater  Strike  Military  Space 

Systems 

Electronic  Combat  Government  Systems 

Operations 


Enterprise  Systems 
and  Security 

iT/Network  Outsourcing 

inteiiigence 


iSR  Sateiiite  Systems 

Missiie  Defense  Sateiiite 
Systems 

MiLSATCOM  Systems 

Environmentai  &  Space 
Science  Sateiiite  Systems 

Directed  Energy  Systems 

Strategic  Space  Systems 


Federai,  State/Locai 
&  Commerciai 

Homeiand  Security 
&  Heaith 
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Navai  Systems  integrator 

Surface  Combatants 

Expeditionary 
Warfare  Ships 

Auxiiiary  Ships 


Systems  Support 

Base  and  infrastructure 
Support 

Range  Operations 
Maintenance  Support 
Training  and  Simuiations 


Marine  Composite  Technical  and 

Technology  Operational  Support 


Coast  Guard  Cutters 
Commerciai  Ships 


Live,  Virtuai  and 
Constructive  Domains 

Life  Cycie  Optimization 


Nuciear  Aircraft  Carriers 


Performance  Based 
Logistics 


Nuciear  Submarines 
Fieet  Maintenance 


Modifications,  Repair 
and  Overhaui  (MRO) 


Aircraft  Carrier 
Overhaui  &  Refueiing 


Suppiy  Chain  Management 

Lead  Support 
integrator  (LSi) 


Long  Legacy  of  High  Maturity 


MORmROP  GRUMMAN 


•  Northrop  Grumman  has  a  long  history  of  embracing  High  Maturity 

-  1986  First  CMM  appraisal 

-  1996  Achieved  first  High  Maturity  assessment  CMM  for  Software 

-  2002  Early  adopter  of  CMMI  High  Maturity  in  2  appraisals 

•  Northrop  Grumman  currently  has  12  CMMI  High  Maturity  Appraisals  (26%  of  all  US 
company  CMMI  Maturity  Level  5  appraisals) 

-  The  Information  Systems  Sector  currently  has  9  of  12  Northrop  Grumman  High 
Maturity  Appraisals  (19.6%  of  US  companies  at  CMMI  Level  5) 

-  The  Defense  Systems  Division  currently  holds  5  of  them  that  cover  27 
development  sites  (11%  of  US  appraised  organizations  at  CMMI  Level  5) 

-  Another  DSD  High  Maturity  appraisal  is  underway  as  we  speak. 

•  There  is  a  reason 

-  Our  Division  General  Manager  has  managed  High  Maturity  organizations  since 
1996 

-  We  firmly  believe  that  we're  better  at  what  we  do  because  of  our  commitment  to 
high  maturity  processes 
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High  Maturity  has  been  a  part  of  our  development  life  for  over  a  decade 


CM  Ml  Benefits  -  Often  Expressed  as  ROI 


MORmROP  CRUMMAM 


2005 


Table  2:  CMMf  Performance  Resitlts  Sirmcnary 


Fei'forinjince 

C  atf  gon 

^fedJ»u 

Einpraveiiienr 

Number  of  Dora 
Points 

Cost 

34% 

29 

Schedule 

5Q\ 

22 

ProQiEcdvity' 

61% 

20 

Qualilj' 

4S“d 

J4 

CustoaQfiT 

Sads^cdoai 

14“  D 

7 

ReTumoQ  luvestaieut 

4.0  1 

1  > 

9 

Performance  Results  of  CMMI -Based  Process  Improvement, 
D.  Gibson,  D.  Goldenson,  K.  Kost,  Aug.  2006 
SEI  Technical  Report 

© 

Performance  Results  From  Process  Improvement, 

SEI  and  DACS,  March  2007,  Software  Tech  News 


© 

2007 


Performance  Category 

Median 

Number  of 
Data 
Points 

Cost 

1 

20% 

21 

Schedule 

37% 

19 

Productivity 

62% 

17 

Quality 

50% 

20 

Customer  Satisfaction 

14% 

6 

Return  on  Investment 

4  7: 1 

I6 
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Benefits  -  Often  Increased  Productivity 


MORmROP  CRUMMAM 


0 

Baseline 

Productivity 

CMMI® 

Productivity 

improvements 

Impact 

Average  Project  Size 

133 

133 

Average  FP/EM 

10.7 

24.8 

+132% 

Average  project  duration 
(months) 

6.9 

3.5 

-50% 

Average  effort/ FP 

$939. 

$467. 

-50% 

Defect  Density 

0.0301 

0.0075 

-75% 

© 

Performance  Outcomes  of  CMMI 
Based  Processess, 

P.  McNoone  &  S.  Rohde, 

Lockheed  Martin 

0 

Improved  Performance  Should  be 
Expected  from  Process  I  mprovement, 
D.  Garmus  &  S.  Iwaniki, 

David  Consulting  Group 
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Software  Unit  Cost 


Pre  NW/IPQ 


Post  NW/IPQ 


Everybody  Does  Defect  Analysis  ...BUT.. 


MORmROP  CRUMMAM 


•  The  cost  of  correcting  defects  does  vary 

•  “Cost  to  correct"  depends  upon  when  you  find  and  fix 


Phase  Hours  to  FixQ 

Requirements  through  Code  /  Build  6 

Component  /  I  &T  Testing  37 

System  &  Acceptance  Testing  74 

Post  Delivery  123 


•  Level  3  organizations  find  defects  later  in  the  cycle 

•  Level  5  organizations  find  defects  earlier 
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Source  of  Data: 

Madachy,  Ray.  "Quantitative  Process  Management  and  Software  Quality  Management 
Department  of  Computer  Science,  University  of  Southern  California,  February  2000 


Distribution  -  When  Defects  Found 


MORTHROR  GRUMMAN 


40.0% 


35.0% 


30.0% 


25.0% 


20.0% 


15.0% 


10.0% 


5.0% 


0.0% 


■z 


Requirements  Design 


Code/Buiid  Component/I&T  Systems-  PostDeiivery 
Test  Acceptance  Test 


CMMI  Level  3  benchmark  %  defectdistribution  ■  CMMI  Level  5  benchmark  %  defect  distribution 
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Real  Value  of  CMMI  High  Maturity 


MORmROP  GRUMMAN 


Cost  of  100  Defects  Found  &  Fixed 

Phase 

Level  3  Org 

Level  5  Org 

Value  of  HIMAT 

%  Fix  Hrs  %  Fix  Firs  Delta  Firs  %  Savings 


Reqs  thru  Code  22%  132  65%  390  (258)  (5%) 

Build  (6  hrs) 

Component/  38%  1406  20%  740  666  14% 

l&T  (37  hrs) 

System  &  32%  2368  12%  888  1480  30% 

Acceptance  Test 
(74  hrs) 

Post  Delivery  8%  984  3%  369  615  13% 

(123  hrs) 


Total  4890  2387  2503  51% 
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50%  Savings  and  Fewer  Defects  Delivered  to  Your  Customers  -  Priceless 


Value  Received  Varies  by  Defect  Rate 


MORTHROR  GRUMMAN 
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Hours  to  Correct  Defects  per  100  KESLOC  -  Level  3  vs  Level  5 


Defects  per  KESLOC 


High  Maturity  Benefits 


Benefit 

•  Improved  Productivity 


•  Fewer  Delivered  Defects 


•  Lower  Cost  of  Defect  Correction 
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MORmROP  GRUMMAN 


Result 

Reduced  Development  Cost 
Shorter  Development  Schedule 
Better  Acceptance  Test  Results 
Better  User  Satisfaction 
Reduced  Development  Cost 
Fewer  Development  Delays 


NORTHROP  GRUMMAN 


The  Measured  Value 

ofCMMI 


M.  Lynn  Penn 

Director  Process  Management 
Lockheed  Martin  Corporation 
Information  Systems  &  Global  Services 

NDIA  -  November  2009 


Challenge 
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To  BALANCE  the  cost  of  implementing 
process  requirements  with  benefits  to  the 
business  and  customer 


Traditional  Advertised  Benefits 


■  Productivity  Increase 

■  Quality  Improvement 
■  Defect  detection 

■  Replanning  -  decrease 

■  ROI  -  average  -  4:1 


Defects  per  KSLOC 


SI  -  Owego 


Quality  and  Productivity 

Software  Defect  Density  Software  Productivity 


1999  2000  2001  2002  2003  2004  2005  2006  2007 

Year 


On-Time  Delivery  (All  Products)  Performance  at  Delivery  (Ail  Product')) 


2004 


2005 


2006 


2007 


Lates 

%  On-Time 


201 

99.11% 


176 

99.14% 


127 

99.29% 


79 

99.59% 


Delivered  (K) 
Defects 
%  Error  Free 


LOG  /  Hour 


IS&GS 


S/W  productivity  increased  52%  over 
baseline  average  capability 


u 
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S/W  cost  decreased  23%  in 
Constant  2004  dollars 


Historical  CMMI  L5 


Customer  Satisfaction 


Overhead  Costs 


Defect  Find  /  Fix  cost  down 
by  21% 


Last  updated;  2006  with  2005  Data 


Agenda 


Realizing  CMMI  Benefits 
Enablers 
Inhibitors 


Enablers 
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Process  I  mprovement 

■  Operated  and  managed  as  a  program 

Process  Architecture 

■  Implementation  of  the  Business  Strategies 

■  Integrated  into  the  Business  Rhythm 

CMMI  -  SVC 

■  Process  as  a  service/  EPG  a  service  organization 

High  Maturity 

■  Benefits  increase  exponentially  with  the  HM  Tools 


Enablers 


People 

Process  Technology 

LMCO  Strategic  Plan 
Business  Unit  Strategic  Plan 
Technology  Plan 
Learning  Development  Plan 
Process  Improvement  Plan 


Inhibitors 


■  MYTHS,  MISCONCEPTIONS,  IGNORANCE 

■  External 

■  Internal 
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External  Inhibitors 

Negotiation  Challenges 

■  Bid  on  accurate  performance  baselines/ 
historical  data 

Let's  try  it  for  this  amount  (out  of  the  blue) 

Try  it  and  MISS 

Problems  arise  -  defects  are  found 

■  Maturity  Level  5  and  have  Defects  ????? 

■  Process  not  a  panacea 


External  Inhibitors  (cont) 
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Required  operation  at  Maturity  Level  3  just 
take  out  that  “other  stuff' 

Teams  composed  with  multiple  level 
companies  -  forced  to  use  highest  level  of 
processes 


Internal  Inhibitors 
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■  Benefits  take  too  long  to  be  realized 

■  Backing  stops  short 

■  Reasons 

Large  Programs 
Many  Programs 
Extended  Life  Cycles 

■  “Price  to  Win"  mentality 

■  Bid  this  to  win  -  regardless  of  performance 
data 


How  to  Survive 


■  Education 

■  External 

Set  and  understand  expectation 

■  Internal 

Executive  knowledge  and  buy  in 

■  Understand  performance  today  predicts  tomorrow 

■  Consistency 

■  Institutionalization  with  tailoring 

■  PERSEVERENCE 
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Effective  Use  of  CMMI® 


NDIA  Position  Paper 

Summary  of  NDIA  industry  position  statements  for 
obtaining  best  vaiue  from  CMMI  investments 


CMMI  WG  -  Overview 
November  2009 
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The  Effective  Use  of  CMMI« 


STRENGTH  TIIKOLGII  WsDLSTRV  &  TEtHNOUXiV 


1.  Good  processes  increase  the  likelihood  of  achieving  successful  project 
performance 

2.  CMMI  is  a  model,  not  a  standard  -  adapt  CMMI  to  your  business  environment, 
resources,  and  objectives 

3.  Focus  on  business  improvement  objectives  -  a  primary  emphasis  on  achieving 
levels  may  not  achieve  significant  benefits  and  may  increase  rather  than 
decrease  costs 

4.  High  maturity  is  a  business  case  -  Justify  the  investment;  many  organizations 
find  business  value  in  improving  processes  even  at  lower  CMMI  maturity  levels 

5.  Maturity  level  ratings  are  not  alone  a  predictor  of  project  performance  -  many 
other  factors  can  be  significant  contributors 

6.  Don’t  specify  maturity  levels  in  acquisitions  -  use  CMMI  to  probe  supplier 
capability  and  process  execution  risks 

7.  Greatest  benefits  of  appraisals  are  from  improvements,  not  evidence  or  ratings  - 
disproportionate  effort  on  appraisal  preparation  risk  can  diminish  business 
returns 


•“The  Effective  Use  of  CMMI®”,  NDIA  Systems  Engineering  Division,  June  2009. 

http://www.ndia.orq/Divisions/Divisions/SvstemsEnqineerinq/Documents/CMMI%2QWorkinq%2QGroup/CMMI%2QNDIA%2Qposition%2Qstatement  final  .pdf 


The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 
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The  Economics  of  CMMI 


STRENGTH  TIIKOIGH  WsDLSTRV  &  TEtHNOUXiV 


Overview: 

•  Developed  by  NDIA  CMMI  Working  Group 

•  Guidance  by  industry,  and  for  industry,  on 
achieving  business  value  through  CMMI 

•  Suggested  CMMI  strategies  and  mechanisms, 
intended  to  be  tailored  much  like  the  model  itself 

Content: 

1.  Guidance  on  achieving  business  performance 
improvement  through  economical  use  of  CMMI 

2.  Guidance  on  effective  CMMI  implementations  to 
address  common  business  issues 

Objectives: 

•  Provoke  thoughtful  dialog  on  the  effective  use  of  CMMI 

•  Influence  the  mindset  of  CMMI  business  value  -  focus  on  improvement 

•  Help  raise  expectations  across  industry  for  results  achieved  through  CMMI 
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The  EcoMitics  of  CMMI 


The  Economics  of  CMMI* 

NDLV  Systems  Engineering  Division 
CMMI  Working  Group 
Version  1.0 
October  15,2009 


http://www.ndia.org/Divisions/Divisions/SvstemsEngineering/Pages/CMMI  Working  Group.aspx 


The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 
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The  Economics  of  CMMI 
-  Summary 


STRENGTH  TIIKOIGH  WsDLSTRV  &  TEtHNOUXiV 


Business  returns  on  CMMI  investments  are  dependent  iargeiy  on 
underiying  principies 

•  Objectives  -  alignment  with  business  goals 

•  Sponsorship  -  leadership,  commitment,  resources 

•  Action  -  improvement  velocity  for  business  needs 

•  Engagement  -  participation,  project  focused 

•  Value  -  performance  results  to  Justify  investments 

•  Motivation  -  performance  improvement  vs.  ratings 

These  factors  are  under  an  organization’s  controi 

•  The  Economics  of  CMMI  is  a  balance  sheet  for  obtaining 
best  value  from  CMMI 

•  Implementation  strategies  govern  whether  CMMI  investments  translate  into 
improved  business  performance,  or  simply  added  costs  of  doing  business 

Focus  on  business  vaiue  to  provoke  thoughtfui  diaiog  and  raised 
expectations  for  the  effective  use  of  CMMI 


The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 
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The  Benefits  of  CMMI 

Case  Study  of  a  Small  Business 
CMMI  Level  5  Organization 

Executive  Panel 

CMMI  9^^  Technology  and  User  Group  Conference 

November  17,  2009 
Denver 


Software  Process  Achievement 

Award 


>  Sponsors 

-  IEEE  Computer  Society 

-  Software  Engineering  Institute  (SEI) 

>  Winners 


ais 


NASA  Goddard  Space  Flight  Center  (94) 
Raytheon  (95) 

Hughes  Aircraft  (97) 

Advanced  Information  Services  Inc.  (99) 

Oklahoma  City  Air  Logistics  Center  (99) 


Wipro  (03) 

IBM  Australia  (04) 

Productora  de  Software  S.A.  (06) 


IComputerI 

SOCIETY 


‘Tor  exceptional  work  in  applying  and  tailoring  tlie 
principles  of  continuous  process  improvement  to 
enhance  quality,  productivity,  and  customer  satisfaction, 
as  well  as  institutionalizing  process  improvement  in  the 
de\^elopment  of  software  and  the  provision  of  consulting, 
training,  and  infrastructure-support  services.” 


Advanced  Information  Services  Inc. 


©AIS  2007 


AIS  CMM/CMMI  Assessment 


History 


Date 

Levels  Assessed 

Levels 

Satisfied 

Assessor 

April  1996 

SW-CMM  Levels  2  &  3 

1 

Jeannie  Kitson 

April  1999 

SW-CMM  Levels  2  to  4 

3 

Jeannie  Kitson 

Nov.  2000 

SW-CMM  Levels  2  to  3 

3 

Jeannie  Kitson 

Nov.  2002 

SW-CMM  Levels  2  to  4 

3 

Inigo  Garro 

Nov.  2004 

SW-CMM  Levels  2  to  4 

4 

Gloria  Leman 

Dec.  2005 

SW-CMM  Levels  2  to  5 

5 

Gloria  Leman 

Dec.  2007 

CMMI  Maturity  Levels  2  to  5 

5 

Ed  Weller 

ais 


Advanced  Information  Services  Inc. 
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©AIS  2007 


SEI  CMMI  Maturity  Level  5 


ais 

The  AIS  Software  Development  Organization 

(Federal  and  Commercial) 
has  successfully  completed  a 

scampi'^  A 

(Standard  CMMI®  Appraisal  Method  for  Process  Improvement) 
and  satisfied  the  goal  requirements  to  achieve  a  rating  of 

CMMI-DEV  vl.2 
MATURITY  LEVEL  5 

December  14,  2007 

as  listed  on  the  Sofbv^are  Engineering  Institute  PARS  webpage 

Edw-ard  F.  Weller 

SEI-Certified  SCAMPI  High  Matiuity  Lead  Appraiser  0000096-00 

*  CMMI  ii  ^pEtBnd  in  tiu  U£.  Patent  ft  TtzidsEcarl  DS»  1^-  Mdloe  UBhmih' 

SC.4MFI  is  a  mn.'XD  maik.  sxf  Caraagif]  Miillm  IMsnixity 


Advanced  Information  Services  Inc. 
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SCAMPI  A  -  Final  Findings 
AIS  Global  Strengths 

>  TSP  coaches  provide  continuous  mentoring  for  project  team 
members 

>  Process  focus  at  all  levels  in  the  organization 

>  Open  communication 

>  Self-managed  team  structure  and  roles 

>  Individuals  with: 

-  Strong  quality  focus 

-  Commitment  to  customer  and  organization 

-  Sense  of  ownership 

>  Opportunity  for  involvement  with  multiple  groups  within  the 
organization 

>  Empowered  to  make  decisions  that  affect  the  organization 

ais _ 
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CMM/TSP  Benefits 
Schedule  Performance 


Schedule  Deviation  individuai  Vaiue  Controi  Chart  - 

Development  Phases 

CMM  PSP/TSP 

350 

300 

250 

200 

I  150 

I  100 

a> 

*=*  50 

0 

-50 
-100 
-150 

Date  of  Project  Phase  Start 
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Advanced  Information  Services  Inc. 
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CMM/TSP  Benefits 


Effort/Cost  Performance 


CU 


Effort  Deviation  Individual  Value  Control  Chart  -  Development  Phases 


CMM 

I 

t 


PSP/TSP 


85% 


■| 


-I — o — 


>10  years  history  of  delivering  within  4%  of 
Q  -  ,  committed  effort/cost  on  average 

§  17%  !”'  "  '  "  ‘  '  "  "  ’ 

o  _ >.‘‘3 _ -s.- 

9  ::  r  ^1 


-tQ) - ^ 


_ lU  a _ Dl^ « HJ  1^2 —  i 


Tfn-TiH-Tf 


01^97  o  OlflSP  0^/99  mm 


WT, 

mm  oiA)3  mm  oia)5  oiyoe  mm 


.1 


Date  of  Project  Phase  Start 


o  Individual  Data  Points 


U  p  p  er  N  atural  P  ro  cess  Li  m  it 


Lovter  Natural  ProcessLimit 


-  One  Standard  Deviation 


ais 


Advanced  Information  Services  Inc. 
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CMMI  Quality  Results 


Source:  The  TSP  in  Practice,  SEI  Technical  Report,  September  2003 


Advanced  Information  Services  Inc. 
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CMM/TSP  Benefits 


ualitv  Performance 


User  Acceptance  Test  Defects  Per  KLOC  -  New  Development  Projects 

(Avg=0.15,  UCL=0.66) 


s 


0.8 


0.6 


0.4 


0.2 


--U£t- 


< 


-0.2 


-0.4 


-0.6 


10  years  history  of  post-delivery  defeets  less 
than  0.15  per  KLOC  on  average 


Avg 


I - 1 - r 


96  01/97  02/97  07/97  11/98  03/00  06/00  12/00  01/01  04/01  05/01  07/01  06/02  05/05 


-Let- 


KLOC  -  Thousand  Lines  of  Source  Code 
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Advanced  Information  Services  Inc. 
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What  If? 


What  If  It  Takes  A  Billion  Lines  Of 
Code  Per  Year  To  Modernize  The 
U.S.  Government  In  Five  Years? 


ais 


Advanced  Information  Services  Inc. 
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U.S.  Government  Modernization 
Maintenance  Cost  -  1 

>  Assume  most  federal  IT  eontraetors  are  CMMI  L2,  L3 

>  Assume  average  5  defeets/KLOC  in  delivered  product 

>  Government  will  find  5  million  defects  per  year  in 
acceptance  test  and  software  use 

>  Assume  average  40  hours  to  find  and  fix  each  defect 

>  Acceptance  test  and  rework  cost  over  five  years  -  $  1 00 
billion 
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Advanced  Information  Services  Inc. 
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©AIS  2007 


U.S.  Government  Modernization 
Maintenance  Cost  -  2 

>  Assume  most  federal  IT  contractors  are  CMMI L5 

>  Assume  average  1  defect/KLOC  in  delivered  product 

>  Government  will  find  1  million  defects  per  year  in 
acceptance  test  and  software  use 

>  Assume  average  40  hours  to  find  and  fix  each  defect 

>  Acceptance  test  and  rework  cost  over  five  years  -  $20 
billion 

>  $80  billion  cost  avoidance  in  five  years 

ais _ 
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What  If  The  Contractors  ?  ( 1 ) 

Bid  firm  fixed  price  development  after 
requirements  phase 

Guarantee  less  than  8%  deviation  in  committed 
schedule 

Guarantee  reduction  in  government’s  acceptance 
testing  time  by  orders  of  magnitude 

ais 


Advanced  Information  Services  Inc. 


13 


©AIS  2007 


What  If  The  Contractors  ?  (2) 

Guarantee  delivered  product  defect  density  of 
less  than  0.3  defects  per  KLOC 


Offer  life  time  warranty  on  defects 


AIS  offers  all  of  the  above 

ais 


Advanced  Information  Services  Inc. 
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Benefits  of  CMMI 

>  Proven  organization  capability  to  deliver 
nearly  defect  free  products  on  predictable 
cost/schedule 


>  Joy  in  work 


ais 


15 


©AIS  2007 


Advanced  Information  Services  Inc. 


Transforming  The  World  Of  Software 

Models  Of  Excellence 


Advanced  Information  Services  Inc. 
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What  does 

“FUN  ON  THE  JOB” 

Mean  to  you? 


Advanced  Information  Services  Inc. 
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Contact  Information 


Girish  Seshagiri 

Advanced  Information  Services  Inc. 

(703)  286  0781 
Email:  girishs@advinfo.net 
Website:  www.advinfo.net 
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Raytheon 

Customer  Success  Is  Our  Mission 


CMMI®  and  Business 
U  Improvement  at 

7  Raytheon 

%  Dave  Tyler 

^  Sr.  Manager 

Enterprise  Process  Effectiveness 
Raytheon  Intelligence  and  Information  Systems 

17  November,  2009 


Copyright  2009.  Unpublished  Work.  Raytheon  Company. 
Customer  Success  Is  Our  Mission  is  a  registered  trademark  of  Raytheon  Company. 


Raytheon  Today 


Raytheon 

Customer  Success  Is  Our  Mission 


$23.2  billion  in  sales  in  2008 
73,000  employees  worldwide 

More  than  8,000  technology-driven  programs 

Locations  in  50  states,  80  countries,  7  continents 

Among  the  top  5  aerospace  and  defense  companies  in 
the  nation 


A  Global  Leader  in  Defense, 
Homeland  Security  and  other 
Government  Markets 


11/24/2009 


Page  2 


Raytheon 


Customer  Success  Is  Our  Mission 


Raytheon’s  Core  Markets 

Expanding  opportunities  to  provide 
innovative  solutions 


Our  Domain  Knowledge  and  Technical  Leadership  Creates 
Expanding  Opportunities  in  Four  Core  Defense  Markets: 


1.  SENSING 


2.  C3I 


3.  EFFECTS 


4.  MISSION  SUPPORT 


Technologies  that 
acquire  data  and 
create  the  information 
needed  for  effective 
battlespace  decisions 


(Command,  Control, 
Communications  and 
Intelligence  Systems) 
Integrated  real-time 
systems  that  optimize 
operational  planning 
and  execution 


Technologies  that 
achieve  specific 
military  actions  or 
outcomes 


Total  life-cycle  solutions 
that  ensure  NoDoubt™ 
performance 


11/24/2009  Pages 


Raytheon 

Customer  Success  Is  Our  Mission 


Raytheon  Businesses 


Space  and  Airborne 
Systems 

El  Segundo,  CA 

\ 

VT 


Integrated 
Defense  Systems 

Tewksbury,  MA 


BD  and  Raytheon 
International  Operations 

Rosslyn,  VA 


Network  Centric 
Systems 

McKinney,  TX 


\ 

vr 

Missiie  Systems 

Tucson,  AZ 


inteiiigence  and 
information  Systems 

Garland,  TX 


IGiobai 

Headquarters 

ss  Waltham,  MA 


Technicai  Services 

Reston ,  VA 


\ 
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Raytheon  Process  Improvement 


Raytheon 

Customer  Success  Is  Our  Mission 


The  way  we  ^ 
develop  products  &  / 

provide  services  / 

/ 


ip^ 


\ 


V 


SM 


\ 


Our  focus  on  No 
Doubt^^  soiutions 
for  the  warfighter 


/ 


R6g 

Rar^heon  Signift  i 


A  criticai 
success 
measure 


Our  primary  mechanism  for 
continuous  improvement 
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CMMI®  Adoption  at  Raytheon 


Raytheon 

Customer  Success  Is  Our  Mission 


Virtually  all  of  the 
engineering 
population  engaged 
with  CMMI® 


Most  divisions  now 
operating  at 
Maturity  Level  5, 
including  two  v1.2 
appraisals  in  last  12 
months 


Early  Adoption  and  Commitment  to  High  Maturity  at  Raytheon 
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Raytheon 

Customer  Success  Is  Our  Mission 


■  Productivity 

-  Systems  Engineering:  14.3%  Improvement 

-  Software  Engineering:  43  -  65%  Improvement  (depending  on  project) 

-  Hardware  Engineering:  25  -  56%  Improvement  (depending  on  discipline) 

■  Quaiity 

-  Systems  Engineering:  requirements  volatility  reduced  by  56% 

-  Software  Engineering:  12%  improvement  in  defect  containment 

-  Hardware  Engineering:  65%  improvement  in  drawing  defect  density 

■  Cost/Schedule 

-  4%  decrease  in  CPI  and  significantly  reduced  variability 

-  5%  improvement  to  on-time  deliveries 

■  Process  Management  Cost 

-  $15M  savings  over  5  years  in  process  infrastructure  cost 

-  5:1  reduction  in  process  guidance  levied  on  business  execution 


Sample  Benefits 


Maturity  Levels  Don’t  Tell  the  Whole  Story 
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Non-Quantitative  Benefits  of  CMMI® 


Raytheon 

Customer  Success  Is  Our  Mission 


■  Establishes  clear  roles  and  responsibilities  for  business 
execution 

■  Documents  a  common  language  across  broad  spectrum  of 
business  functions 

■  Focuses  process  improvement  on  quality  and  performance 
objectives 

■  Provides  for  structured  decision  making  instead  of  “seat  of 
the  pants” 

■  Helps  answer  the  “are  we  there  yet?”  process 
institutionalization  question 

■  Expands  the  reach  of  Raytheon  Six  Sigma 

■  Changes  expectations  (and  behavior)  of  management 


Achieving  the  “I”  in  CMMI® 
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Looking  Ahead 


Raytheon 

Customer  Success  Is  Our  Mission 


■  Continued  focus  on  High  Maturity 

-Leverage  established  best  practices  in  the  company  to 
enable  high  maturity  practices  across  the  board 

-Invest  in  Capability  Levels  where  Maturity  Levels  don’t 
make  sense 

■  Implementation  of  CMMI-SVC 

—A  visibly  better  “fit”  for  the  engineering  services  business 

-Provides  a  new  benchmark  in  an  increasingly  competitive 
market 


■  Integration  with  other  improvement  paradigms 
—  ISO9001  and  AS9100  in  development  organizations 
-ITIL,  IS020000  and  COBIT  in  services  organizations 
-Lean  models  in  production  organizations 

Steady  Momentum  for  CMMI®  Implementation 
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The  Bottom  Line 


Raytheon 

Customer  Success  Is  Our  Mission 


'^Common  Process^ 
for  Reachback  Across 
Work  Locations 


One  set  of  Processes 
to  Address  External 
Requirements 


Clear  Functional 
Roles  and 
Responsibilities  for 
Program  Execution 


Repeatable 
Process  to  Drive 
Productivity 


Leverage 
Xompany  Investment 
through  Process 
Asset  Reuse 


Centralized 
Process 

Management  ^ 


CM[J-07l418-13a 


Enterprise-Wide 
Process  Certifications 
as  a  Competitive 
Discriminator 


Effective  and 
Efficient  Process 
Deployment  to 
Programs 


CMMI®  Drives  Business  Vaiue  for  Raytheon 
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NORTHROP  GRUMMAN 


Publicly  Reported  Data 
Begging  for  Analysis 

Data  Gathered  From  SEI  PARS  Site 

Everyone  Can  Do  Their  Own  Analysis 

November  17,  2009 

Hal  Wilson 

Director,  Engineering 
Defense  Systems  Division 
Northrop  Grumman  Information  Systems 


©  2009  Northrop  Grumman  Corporation.  All  rights  reserved.  (Log  #  DSD-09-56) 


Overheard  Over  the  Years 


MORmROP  GRUMMAN 


•  Bigger  countries  with  big  economies  are  the  principal  players  in 
CMMI 

-  Smaller  countries  aren't  really  participating 

•  Big  US  companies  with  lots  of  DoD  business  dominate  the  US  CMMI 
arena 

•  DoD  companies  only  apply  CMMI  to  get  the  ratings  for  Government 
competition 


MORmROP  CRUMMAM 


What  the  Publicly  Reported  Data  Says 

(thru  Nov  14,  2009) 


Vast  Majority  are 
CMMI-DEV 

As  of  Nov  14, 

2808  CMMI-DEV 
4  CMMI-ACQ 
0  CM  Ml -SVC* 

2812  Total 

*(CMMI-SVC  only 
eligible  for  Class  A 
appraisals  on  Aug  31.) 

Appraisal  List 
includes  65 
countries: 


©  Ten  of  the  top  20 
may  not  be  the 
countries  you 
expected 


Distribution  of  Appraisals  by  Country  -  Top  20 


■  china 

■  USA 

■  India 

■  Spain 
a  Japan 

■  Brazil 

■  Taiwan 

■  France 

*  South  Korea 

■  Mexico 

■  Argentina 

■  United  Kingdoj 
^  Malaysia 

■  Tltailand 
Canada 

■  Germany 
Italy 

*  Pakistan 
Chile 
Columbia 


Publicly  Reported  Appraisals  (thru  Nov  yt*") 


MORmROP  CRUMMAM 
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CM  Ml  Appraisals  Since  2006 


2006 


2007 


200S 


2009 


■  IQ  mlQ  s3Q  ■4Q  ■  Total 


Note:  3’'^  Qtr  2007  was 
an  anomaly! 

CMMI  Steering  Group 
initiated  the  3  year 
validity  rule  for 
appraisals  and  it  went 
into  effect  on  Aug  31, 
2007 

Slower  growth  in  2008 
may  have  been  the 
result  of  older 
'appraisals  for  life'  being 
reappraised  in  2007. 

2009  is  on  pace  to 
break  1000  for  the  first 
time  but  needed  590 
just  to  stay  even 

2010  will  need  883 
to  stay  even 


We've  Had  Steady  CMMI  Growth  Over  4  Years 


Quantity  of  Appraisals  by  Maturity  Level  (Jul  05 -Nov  14, 00] 


Obviously  a  good  thing 


BUT.... 


Only  Maturity  Level  3 
is  growing  robustly 

Maturity  Level  2  is  growing 
modestly 

Maturity  Level  4  is  declining 
slightly 

Maturity  Level  5  is  declining 
-Only  slightly  higher  than  in 
July  2005 

Not  Given  results  are  steady 


Only  Level  3  Appraisals  Increasing 


MORmROP  CRUMMAM 


Why  the  Maturity  Level  3  Surge? 


MORmROP  GRUMMAN 


•  Chinese  Government  announced  stipend  for  achieving  a  CMMI  Maturity 
Level  3  in  2007 

-  A  surge  of  Chinese  appraisals  ensued  and  continues 

-  China  now  has  the  largest  quantity  of  appraisals  of  any  country  in  the  world 

•  CMMI  Steering  Group  eliminated  the  “appraisal  for  life",  creating  a 
surge  in  appraisals  in  2006-2007 

-  There  still  is  a  regular,  growth  overall  even  though  ~  1/3  of  appraisals  now 
retire  each  year 

-  144  appraisals  are  needed  each  year  just  to  maintain  the  2006  level. 

•  In  2008,  SEI  initiated  High  Maturity  Audits  without  clearly  stated 
appraisal  criteria 

-  Some  companies  re-appraised  at  Level  3  until  they  could  predict  that  their 
expenditures  would  yield  the  desired  results 

•  As  a  result,  some  did  not  pursue  High  Maturity  appraisals  until  the 
criteria  was  published  in  Dec  08. 

-  Hi  Mat  appraisal  rates  versus  Level  2-3  have  been  dropping  each  year 

-  From  11%  to  6.5%  to  5.1%  since  August  2007 


External  Influence  or  Early  Adopters? 


MORmROP  GRUMMAN 
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Growth  in  Appraisals  (jui  zoos  to  nov  14,2009) 


Average  %  Growth  =  834% 


China 

USA 

India 

Japan 

Taiwan 

South  Korea 

Malaysia 

Thailand 

Mid  East 

South  America 

Europe 


flffl 


= 2385% 
=  624% 
=  491% 
=  231% 
=  1233% 
=  971% 

=  3750% 
=  755% 
=  920% 
=  2714% 
=  518% 


China  USA  India  Japan  Europe  Taiwan  South  Malaysia  Thailand  Mid  East  South 

Korea  America 

■  Jul-05  ■Nov-09  ■  Growth 


2005  Early  Adopters 


USA 

116 

India 

57 

Japan 

40 

Europe 

39 

China 

35 

2009  Leaders 

China 

835 

USA 

624 

India 

280 

Europe 

366 

Japan 

93 
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MORmROP  CRUMMAM 


Rate  of  High  Maturity  Appraisals  Slowing 

(as  of  Nov  7,  2009) 


2006  Events  that  may 
have  had  impact 


OSD  Director  of  SSE 

•  Questioned  the 
'appraisal  for  life'  status 

•  Claimed  that  Level  5 
companies  weren't 
demonstrating  their 
Level  5  credentials 

•  Some  complained 
about  the  quality  of 
High  Maturity  appraisers 


CMMI  SG: 

1.  Instituted  the  3  year 
validity 

2.  Instituted  Hi  Mat 
appraiser  training  & 
authorization 


Period  of  Measurement  from  Sep  thru  Aug 


May  be  some  hope  in 
2010 


Events  That  Influenced  Hi  Mat 


MORTHROR  GRUMMAN 


Factor:  Companies 
have  combined 
operations  to  save 
costs 
BUT  ... 

Other  events  have 
impacted  CMMI  Hi  Mat 
Appraisal  Rates! 

1)  CMMI  SG  instituted  a 
3  year  appraisal  life 

2)  SEI  instituted  an 
undocumented  audit 
process  for  all  Hi  Mat 
appraisals 

3)  Uncertainty  about  how 
DoD  viewed  Hi  Mat 
appraisal  holders 

Those  up  for  reappraisal 
may  have  wondered  if  it 
was  worth  it  to  continue 
if  their  integrity  was 
questioned  and  they 
weren't  sure  of  the  rules 


Distribution  of  Hi  Mat  Appraisals  (juiy2oo5) 


MORTHROR  GRUMMAN 


Appraisal  and  Hi  Mat  Distribution  by  Region 


Asia  Indian  North  Europe  Central  &  Africa  &  Australia, 

Subcontinent  America  South  Middle  East  New  Zealand, 

America  Malaysia  & 

Indonesia 


July  2005 

%  Hi  Maturity  (Level  4  &  5) 

WW  27.3% 

North  America  28.3% 


Note: 

Appraisals  did  not  begin 
retiring  untii  2007. 

Surge  of  CMMi  appraisais  from 
early  CMMI  adopters  in  2002-3 
were  stili  included  in  the  totals 

Biased  the  percentages  during 
the  eariy  adopter  period  from 
2002  to  2005 


Distribution  of  Hi  Mat  Appraisals  (mruNovu) 
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Appraisal  &  Hi  Mat  Distibution  by  Region 


Asia  North  Indian  Sub  Europe  Central^  Afrlca&  Australia, 

America  Continent  South  Middle  East  New 

America  Zealand, 

Malaysia  & 
Indonesia 


Hi  Maturity  (Level  4  &  5) 
Average  7.2%  of  WW 
appraisals 

(down  from  34.9%  in  J  uly 
2005) 

India  averages  20.1%  (Down 
from  36.5%  in  2005) 

North  America  averages  7.2% 
US  averaged  7.0%  (Down 
from  28.3%  in  J  uly  2005) 

Canada  averages  13.9% 
Mexico  averages  8.4% 

China  averages  6.2%  (Down 
from  7.5%  in  J  uly  2005 
South  Korea  averages  (10% 
Up  from  2.1%  in  2005) 

Note:  Appraisals  prior  to  Oct 
2005  (114)  are  not  included 


Pursuit  of  Maturity  Level  5  Impacted  by  US  DoD? 

(113  of  156  organizations  (72.4%)  don't  seem  to  be  affected) 


MORmROP  CRUMMAM 


High  Maturity  Appraisals  by  Country  (Nov  14^  2009) 


India  China  US  Europe  South  Korea  McKico  Canada  Japan  Phitipines  Mid  East  Malaysia  Taiwan  Vietnam 

America 


■  Level  4  ■  Level  5  ■  Hi  Mat  ■  HI  Mat%  of  Appraisals 


Philippines,  India,  Canada,  Korea,  and  Mexico  still  show  more  than  10%  Hi  Mat  appraisals. 
India  maintains  close  to  a  20%  Hi  Mat  appraisal  percentage. 


18  US  Companies  Remain  CMMI  Maturity  Level  5 

(Nov  14,  09  Compared  to  "^8  companies  &  2  DoD  in  Jul  2005) 


MORmROP  CRUMMAM 


US  Companies  with  CMMI  Maturity  Level  5  Appraisals 


★  ■  Moithrop  Grumman, 

★  ■Raytheon 

■  General  Dynamin 

★  IBAE 

■  Lockheed  Martin 

■  Boeing 
bSAIC 


These  8 

2005  Companies 
Account  for  71.8% 
of  Current  US 
Maturity  Level  5 
Appraisals 


★  iCSC 

II L-3  CommunicadDns 

■  Advanced  Info  Services 

■  Avktd.  !nc. 

V  ORSTechnotjpgEes 

■  EDS 

■  Green  HilEs  Software 
IBM 

■  Keane 


After  all  these 
years, 

why  do  they 
still  believe 
there  is  vaiue 
in  High 
Maturity? 


Overwatt h  Systems  LTD 


•  Souttiwest  Research  Institifte 


Telephonrcs  Corp 


Maybe  these  companies  use  measurement  and  analysis  to  prove 

Hi  Mat  value  for  themselves! 


Possible  Explanations? 


MORTHROR  GRUMMAN 


(Even  with  consolidation  of  organizations,  MLS  quantities  have  remained  about  at  2006  levels.) 


Could  it  be  that: 

•  Companies  with  successfui  High  Maturity  methods  won't  abandon 
them  if  they  are  working? 

-  They  believe  that  there  is  value  in  managing  with  fact  instead  of 
supposition 

-  They  measure  the  value  of  their  improvements 

•  Our  VP  &  General  Manager  won't  allow  business  units  to  manage  without  data 

•  He  uses  that  style  of  management  and  won't  relax  his  standards  of  performance 

•  His  Division  has  grown  from  $450  M  in  2002  to  almost  $  4  B  in  2009. 

•  Perhaps  the  companies  who  were  going  for  maturity  leveis  instead 
of  improvement  never  learned  to  get  the  reai  vaiue  from  High 
Maturity? 

-  That  may  be  a  reason  some  companies  have  not  gone  through  Hi  MAT 
reappraisals 


Pursuit  of  High  Maturity  seems  to  depend  upon  perseverance  and  dedication. 
Expecting  to  achieve  big  benefits  without  continued  effort  and  dedication 

doesn't  appear  to  pay  off! 


Your  Challenge  Awaits 


MORmROP  GRUMMAN 


•  First  look  at  the  data  and  do  your  own  analysis 

-  See  how  you  stack  up  against  your  competitors  across  the  world 

•  Validate  your  impressions  and  look  into  potential  factors 

-  First  determine  if  what  others  are  saying  has  any  validity  in  fact 

-  Don't  be  afraid  to  ask  why  the  traditional  perceptions  don't  seem  to  be  valid 

-  Most  importantly,  look  for  what  matters  to  your  organization 

•  Keep  your  eye  on  CMMI-ACQ  and  CMMI-SVC 

-  It  would  be  natural  to  see  if  new  organizations  or  existing  organizations 
predominate  initial  adoption 

-  It  may  be  that  your  competitors  will  adopt  these  new  CMMI  constellations 
to  improve  their  operations 
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Benefits  to  the  Evolution  of  High  Maturity 
Software  Development:  A  15  Year  Case  Study 


Daniel  W.  Drew,  Erik  Likeness 
United  Space  Aiiiance,  LLC 


Copyright  ©  2009  by  United  Space  Alliance,  LLC.  These  materials  are  sponsored  by  the  National 
Aeronautics  and  Space  Administration  under  Contract  NNJ06VA01C.  The  U.S.  Government  retains  a 
paid-up,  nonexclusive,  irrevocable  worldwide  license  in  such  materials  to  reproduce,  prepare,  derivative 
works,  distribute  copies  to  the  public,  and  perform  publicly  and  display  publicly,  by  or  on  behalf  of  the 
U.S.  Government.  All  other  rights  are  reserved  by  the  copyright  owner. 
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United  Space  Alliance 


Agenda 


*  A  brief  history  of  the  Fiight  Software  Eiement  (FSWE) 

*  Overview  of  the  FSWE  software  process 

*  Changes  to  High  Maturity 

*  Changes  to  the  FSWE  software  process 

*  CMMI  as  a  Vehicle  to  Meet  Customer  Needs 

*  Benefits  to  USA,  our  Customers,  and  Software  Acquisition 


US^ 
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United  Space  Alliance 


Functional  Structure  of  Onboard  Software 


Preflight 

Initialization 

Checkout 


Ascent / 
Abort 


On-orbit 
Systems  and 
Payioad 

yy  Management 


Mass 

Memory 

Utiiity 
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United  Space  Alliance 


Continuous  Process  Improvement  Started  in  1976 


Oversight  analysis 


Reliability  m odeling 


LAN  based  developm  ent 
env  ironm  ent 


1977 

1978 

1979 

1980 

1981 

1982 

1983 

1984 

1985 

1986 

1987 

1988  1 

1  1989  1 

1  1990 

1991 

1992 

1993 

1994 

1995 

¥  Recognized  as  CMM 


US^ 
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United  Space  Alliance 


Evolution  of  Capability 


*  The  FSW  Organization  has  practiced  the  eiements  of  high  maturity  for 
over  20  years. 

*  From  a  quaiity  perspective,  our  understanding  of  common  and  special 
cause  variation  within  our  processes  has  aiiowed  us  to  optimize  our 
quality  to  a  world-class  level 


US^ 
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United  Space  Alliance 


Evolution  Cont. 


*  Today  there  are  three  major  business  areas  in  the  FSW  Organization 

-  Reai-Time  Human  Rated  Software 

*  Deveiops  On-Board  Guidance/Navigation/Controi/Support 
Systems  software  for  the  Space  Shuttie 

-  Mission  Critical  Application  Tools 

*  Develops  ground  support  software  simulation/testing 
environments  for  Human-Rated  Shuttle  Software  Validation  and 
Mission  Support  Activities 

-  Avionics  Integrated  Laboratory  Support  Software 

*  Develops  ground  support  software  simulation/testing  which  are 
integrated  with  Shuttle  Hardware  to  Validate  Human-Rated 
Shuttle  Software  and  Mission  Support  Activities 

*  Our  CMM/CMMI  ratings  began  with  Human-Rated  Software  and  evolved 
over  the  years  to  include  all  three. 


US^ 
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High  Maturity  Evoiution:  Modei  Changes  L4 


CMM 

CMMI  1.1 

CMMI  1.2 

Quantitative  Process 
Management 

Quantitative  Project 
Management 

Quantitative  Project 
Management 

Control  the  process 
performance  of  the  project 
auantitativelv 

Quantitative  management 
clarified  to  center  around 
statistical  techniques 

Focus  on  the  use  of 
performance  baselines  and 

models  in  active  proiect 
management 

Software  Quality 
Management 

Organizational  Process 
Performance 

Organizational  Process 
Performance 

Define  quantitative  quality 
goals  for  project  products 
and  achieve  those  qoals 

Establish  performance 
baselines  and  models  for  the 
organization's  standard 
process 

Focus  on  PPB  and  PPM  tje 
to  business  obiectives  and 
use  of  statistical  techniques 

USA 
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United  Space  Alliance 


High  Maturity  Evoiution:  Modei  Changes  L5 


CMM 

CMMI  1.1 

CMMI  1.2 

Defect  Prevention 

Causal  Analysis  and 
Resolution 

Causal  Analysis  and 
Resolution 

Identify  the  cause  of 
defects  and  prevent  them 
from  recurring 

Amplification  on  causal 
analysis  and  resolution 

activity 

Tie  causal  analysis  and 
resolution  to  a 
“quantitatively  manaqed” 

process 

Technology  Change 
Management 

Organizational  Innovation 
and  Deployment 

Organizational  Innovation 
and  Deployment 

Identify  new  technoloqies 

and  transition  them  into  the 
organization 

Process  Change 
Management 

Improve  the  software 
processes  with  the  intent  of 
improvinq  software  quality 

Select  and  deploy 
incremental  and  innovative 
improvements  that 
measurably  improve 
processes  and  technology 

Tie  to  PPB  and  PPM  and 

Statistical  techniques 

Improvements  show 
measurable  statistical 
siqnificance 

Tie  to  business  obiectives 

USA 
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Evolution  of  the  Flight  Software  Organization 


Man-Rated  Real  Time  Software 


Included  Mission  Critical  Application  Tools  and  Avionics  Integration  Support  Software 


-Formal  Processes 
'Process  Enactment 
■Quality  Management 
■Error  Analysis 
-Process  Analysis 


-Strengthened  all  high  maturity 
practices  by  standardizing 
measurements  across  all 
business  areas  (OPP) 

-Transformed  measurements  into 
metrics  (OPP) 

-Integrated  PPB  and  PPM  (OPP) 

-Formalized  Causal  Analysis 
Process  (CAR) 

-Structured  incremental  and 
innovative  improvements  to  tie  to 
PPB,  PPM  and  Statistical 
techniques  using  Lean  Six  Sigma 
(OID) 


Multi-contract  environment  drives  additional 
emphasis  at  organizational  level 


-Embedded  L6S  DMAIC  Methodology 
into  CAR/OID/OPP. 

-Quantitative  Business  Objectives  with 
detailed  measurement  plans  and  link  to 
PPB  and  PPM.  (OPP) 

-Closed  the  loop  between  QPM  and 
OPR  (CAR) 

-New  or  Changed  PPB  Integrated  into 
Organizational  Assets  (CAR) 

-Quantitative  Business  Objectives  with 
direct  linkage  to  Innovation  projects. 
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spaaisi  jaujo)sno  Improvement 


ustomer  Needs  Closed  Loop  with  PPB  &  PPM 


Mea^y 


SMART  Objectives  Enable  Focus  for  OFF 

j^ecific,  Measureable,  A^chievable,  R^ealistic,  Time-Bound) 


Total  Inserted  Errors 

Individuals  Chart 


OPP:  Process 
Performance 
Baselines  & 
Models 


(A 

o 

m 

% 


-Actual  - Mean  (hist)  - UCL(hlst)  - LCL(hist)  —^Predicted - UPl - LPI 


OPP:  Process 
Performance 
Baselines  & 
Models 


Change  Instrument  (Task) 


CAR  &  OID  Facilitate  Measurable  Improvements 
that  are  Controlled  Quantitatively 


Mea^u 


ilL 


7  7 
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Overall  Benefit 


*  Process  improvement  driven  more  by  changes  in  business 
environment  rather  than  changes  in  the  CMMI 

-  In  the  past,  we  have  had  oniy  a  singie  customer  where  Quality  was 
paramount. 

-  Today’s  Market,  as  well  as  future  market.  Cost  is  becoming  equally 
as  important. 

*  Changes  to  the  CMMI  provided  more  focus  direction  for  appiying  high 
maturity  to  the  organization’s  business  needs 

*  Whiie  the  improvements  we  have  made  benefited  the  market  of  their 
time,  those  methodoiogies  can  be  transiated  to  heip  provide  customers 
with  overall  best-value 

-  Tailor-able  Cost  and  Quality  given  the  needs  of  the  customer 

-  Processes  with  historicaiiy  proven  capabiiity 


US^ 
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Acquisition  Strategy 


*  A  High  Maturity  Organization  shouid  be  abie  to  provide 

-  Reliabie  and  predictable  quality 

*  Tailored  to  your  specific  capability  needs 

-  Substantiated  cost  with  the  ability  to  optimize 

*  Tailored  based  on  your  dynamic  budget 

-  Consistent  predictable  results 

*  Do  not  rely  merely  on  the  CMMI  rating 

-  Look  at  the  application  of  PPBs  and  PPMs 

-  Review  the  organization’s  business  objectives  and  benchmarks 
against  those  objectives 


US^ 
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BACKUP  CHARTS 
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Comparison  of  CMM  and  CMMI  Goals 


Quantitative  Process  Management 

Quantitative  Project  Management 

The  quantitative  process  management 
activities  are  planned. 

The  project  is  quantitatively  managed  using 
quality  and  process-performance 
objectives. 

The  process  performance  of  the  project's 
defined  software  process  is  controlled 
quantitatively. 

The  performance  of  selected  subprocesses 
within  the  project’s  defined  process  is 
statistically  managed. 

The  process  capability  of  the  organization's 
standard  software  process  is  known  in 
quantitative  terms. 

USA 


11/18/2009 

Copyright  ©  2009  by  United  Space  Alliance,  LLC 
Page  14 


United  Space  Alliance 


Comparison  of  CMM  and  CMMI  Goals 


Software  Quality  Management 

Organizational  Process  Performance 

The  project’s  software  quality  management 
activities  are  planned. 

Select  the  processes  or  subprocesses  in 
the  organization’s  set  of  standard 
processes  that  are  to  be  included  in  the 
organization’s  process-performance 
analysis 

Measurable  goals  for  software  product 
quality  and  their  priorities  are  defined. 

Actual  progress  toward  achieving  the 
quality  goals  for  the  software  products  is 
quantified  and  managed. 

USA 
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Comparison  of  CMM  and  CMMI  Goals 


Defect  Prevention 

Causal  Analysis  and  Resolution 

Defect  prevention  activities  are  planned. 

Root  causes  of  defects  and  other  problems 
are  systematically  determined. 

Common  causes  of  defects  are  sought  out 
and  identified. 

Root  causes  of  defects  and  other  problems 
are  systematically  addressed  to  prevent 
their  future  occurrence. 

Common  causes  of  defects  are  prioritized 
and  systematically  eliminated. 
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Comparison  of  CMM  and  CMMI  Goals 


Organizational  Innovation  and 
Deployment 


Technology  Change  Management 


Incorporation  of  technology  changes  is 
planned. 

Process  and  technology  improvements, 
which  contribute  to  meeting  quality  and 
process-performance  objectives,  are 
selected. 

New  technologies  are  evaluated  to 
determine  their  effect  on  quality  and 
productivity. 

Measurable  improvements  to  the 
organization’s  processes  and  technologies 
are  continually  and  systematically 

deployed. 

Appropriate  new  technologies  are  transferred  into  normal  practice  across  the  organization 


Process  Change  Management 


Continuous  process  improvement  is  planned. 

Participation  in  the  organization’s  software  process  improvement  activities  is 
organization-wide. 

The  organization’s  standard  software  process  and  the  projects’  defined  software 
processes  are  improved  continuously. 
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Criteria  for  Audits  of  CMMi  High  Maturity  Appraisais 


*  The  SEI  is  currently  performing  audits  of  all  CMMI  High  Maturity 
appraisals.  The  following  are  the  criteria  being  used  for  these  audits. 
These  criteria  in  no  way  limit  the  application  of  the  model  or  its  intent 
or  judgments  made  during  an  appraisal,  nor  do  they  relieve  the 
organization  from  fully  implementing  the  model. 

*  As  defined  in  the  SCAMPI  v1.2  Method  Definition  Document  Section 
1.1.3,  these  criteria  refers  to  the  instantiations  in  the  representative 
sample  that  are  identified  as  either  focus  projects,  non-focus  projects, 
or  other  organizational  level  instantiations  with  a  scope  that  includes 
the  high  maturity  process  areas. 


US^ 
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Audit  Criteria 


*  Organizational  Process  Performance 

-  (SP  1.1)  Show  the  relationship  between  the  business  objectives  and 
the  processes  seiected  for  process  performance  analysis. 

-  (SP  1.2)  Show  the  anaiysis  and  rationale  for  deciding  what  data  to 
inciude  in  the  process  performance  anaiysis. 

-  (SP  1.3)  Show  the  relationship  between  business  objectives  and 
Quality  and  Process  Performance  Objectives  (QPPOs). 

-  (SP  1.4)  Describe  Process  Performance  Baselines  (PPBs)  in  terms 
of  centrai  tendencies  and  variation  for  the  processes  selected  for 
analysis. 

-  (SP  1.5)  Describe  at  ieast  one  Process  Performance  Modei  (PPM)  in 
terms  of  the  processes  included,  the  controllable  inputs  and  the 
predicted  outputs.  The  model  must  be  statistical  or  probabilistic  in 
nature  rather  than  deterministic,  i.e.,  the  model  considers 
uncertainty  and  predicts  that  uncertainty  or  range  of  values  in  the 
outcome. 
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Audit  Criteria 


*  Quantitative  Project  Management 

-  (SP  1.2)  Describe  how  the  projects  created  their  defined  process  by 
using  PPBs  and/or  PPMs  to  predict  the  ability  of  the  processes 
selected  to  meet  the  project’s  QPPOs. 

-  (SP  1.3)  Describe  the  project’s  rationale  for  selecting  subprocesses 
to  be  statistically  managed. 

-  (SP  1.4)  Show  how  at  least  one  project  used  process  measures  as 
inputs  to  a  PPM  used  to  activeiy  manage  the  project. 

-  (SP  2.2)  Show  that  at  least  one  project  applied  statistical  methods 
to  identify  and  remove  speciai  causes  of  variation  from  seiected 
subprocesses. 

-  (SP  2.3)  Show  how  projects  monitor  the  capability  of  seiected 
subprocesses. 
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Audit  Criteria 


*  Causal  Analysis  and  Resolution 

-  (SP  1 .2)  Demonstrate  that  at  ieast  one  of  the  defects  or  problems 
selected  for  analysis  was  related  to  a  quantitatively  managed 
process,  where  “quantitatively  managed”  is  as  defined  in  the 
glossary. 

*  Organizational  Innovation  &  Depioyment 

-  SP  2.3)  Demonstrate  that  the  effects  of  at  ieast  one  improvement 
were  measured  for  statisticai  significance. 
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Changing  Behavior: 


The  key  to  adoption  of 
compiex  process  technoiogy 


Dr.  Gene  Miluk 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

November,  2009 


Software  Engineering  Institute 


Carnegie  Mellon 


©2009  Carnegie  Mellon  University 


My  goals  for  this  presentation 


1)  Present  new  or  different  approaches  to  technology  transition 

2)  Challenge  your  current  thinking  (changing  change  agents  is  hard) 

3)  Describe  what  I  see  is  working  in  the  field  (and  my  thoughts  on  why) 

4)  Focus  on  the  potential  benefits  to  you  and  your  organization 
inherent  in  these  approaches  to  change 

5)  Describe  my  reactions  and  internalization  of  the  approaches 
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Topics 


>Current  SEI  Change  Management  Approach 
>What's  Needed 
>A  New  Approach 
>Bandura  Social  Learning 
>Bayesian  Belief  Networks 
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Comprehensive  System  Change  Model  (IDEAL) 


Typical  Organization  Structure 


Staffing  the  Process  Infrastructure 


A  Process  Improvement 
Infrastructure 

Core  T earns  are  typically  formed  and  given  responsibilities 
and  roles  for  managing,  facilitating,  and  implementing  a 
change  effortfrom  start  to  finish. 


•  ••• 

TTTT 

Facilitators 

Hid 

Doers 

Management 

Engineering 

Technical 

Steering 

Process 

Woilcing 

Group 

Group 

Groups 

MSG 

EPG 

TWG 

The  Process  Change  Method 


O  Organize  and  Prepare 
Q  Conduct  Organizational  Scan 
©  Establish  Technical  Working  Groups 
©  Understand  Project’s  Current  State 
©  Redesign  the  Process 
O  Develop  Solution 
©  Conduct  Pilot(s)  and  Evaluate 
©  Facilitate  Organizational  Learning 


o 

o 

□ 

A 


Chief  Executive 
Executive  Management 
Middle  Management 
Technical  Staff 


L  EPG  Leader 
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The  IDEALSiw 
Model 


Learning 


SEI IDEAL^*^  Model 


Analyze  n  -  ... 
and  /  "  X 

Validate  ^implement 
- Solution 


Refine 

Solution 


^Charter 

Infra¬ 

structure 


Pilot/Test 
I  Solution 


Iharacterize 
Currents 
L  Desired 
\  States  ^ 


Create 

Solution 


Develop 

Recommen- 

\daiions 


Plan 

Actions 


Develop  ^ 
Approach 


Stimulus  for 
Change 


IDEAL  is  a  service  mark  of 
Carriegie  Mellon  University. 


Propose 

Puture 

Actions 


Build 

Sponsor 

ship 


Set 

Context 


Initiating 


Diagnosing 


Acting 


Establishing 


Based  on  Org  Change  Principles: 

Action  Research 

Socio-tech  Systems 

Plan  Do  Check  Act 

Cascading  Sponsorship 

Parallel  learning  Structures  (SEPG) 


My  experience  with  using  IDEAL: 

•Takes  too  long  (SEI  time  to  move  up) 

•Costs  too  much 

•Engineers  don’t  embrace  it 

•Hard  to  sell  Management  Value  Proposition 


All  |2iH12to  PrssentJ 
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The  assimilation  gap  is  the  gap  between  the  objective  and  the 

deployment 


Assimilation  Gap 

1)  Implementation 

gap 

2)  Performance 

gap 


Robert  G.  Fichman,  Chris  F.  Kemerer,  “The  Illusory  Diffusion  of  Innovation  :  An  Examination  Of 
Assimilation  Gaps”,  Working  Paper  Series  No.746,  Katz  Graduate  School  of  Business,  University  of 

Pittsburgh,  November  1995. 
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Interested  In  ? 


A  streamlined  transition  approach  that  provides: 

•  Compelling  Management  Value  Proposition 

—  Predictable  Costs 
—  Creeping  Commitment 
—  Quick  results  with  measurable  ROI 

•  Concentrated  and  Focused  process  investments 

•  Accelerated  Learning  Environment 

—  New  Processes,  New  Experiences,  New  Data, 
New  Beliefs,  New  Behaviors 

•  Rapid  Predictable  Organizational  Adoption 

•  Continually  Measurable  Results 
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Major  Differences  in  Approach  to  Transition 


•Concentrated  Process 

>  Comprehensive  Packaged  Operational  System  of  Integrated 
Processes 

>  Proven  Performance 

> Integrated  Operational  Measurement  System  (Individual  level) 


•  Focused  Implementation  Strategy 
>Unit  oriented  (Project/Team) 

>JIT  Concentrated  3  level  Training 
>Accelerated  Learning  Laboratory 
> Effective  Project/Team  Launch  Process 
>Coaching  and  continued  support 
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Comprehensive  HP  Development  Process 


Produce 

Requirements 

Specifications 


The  process  elements  are  adapted  to 
the  organization’s  process. 
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Effective  Project/Team  Launch  Process 


TS 


P  Process  Structure 


.  The.TSP  . process  . elements  can  be. : . 

organ  ized  i  nto  wh  atever  process 
■■  stru  ctu  re  m  aRes  Ibe  hi  dst  bu  si  h  ess  ■ 
and  technical  sjense 


The  phases  can  be  implemented  ; 
iteratively  in  small  cycles,;  in  a  spirat 
with  Increasing  cycle  content,  or  : 
sequentially  as: in  a  waterfall,' 


TSP  projects  can  start  on  any 
phase  or  any  cycle 


Each  cycle  starts  with  a  launch  or 
re-lauhch  and  ends  with  a 
postitiortem 
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Operational  Plans 
Implemented  Processes 


Measurement  Framework 


The  TSP  Development  Process 


J 

Hlgh'Lsvvl 

1 
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Resource  Tracking 
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Focused  Implementation:  Building  Organizational 
Capability  Project-by-Project,  Team-by-Team 


Training  Launch 


Corporate 


EPG  identifies  gaps 
and  potential 
improvements,  and 
executes  improvement 
strategies 


Divisions, 
Departments, 
or  Groups  (4) 


Projects  (20) 
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Does  it  work  for  Organizations? 


T5P  Implements  CMMI  -2 


An  o.ipanlz3llQn  using  TSP  has  ijlrecUy 
adaressed  or  ln>p^riwnt?d  most. . . . 
specific  practices  (SP), 
e5\ofSP5  ai  MUS 
ot  Bl 

54*i  ot  SPe  ai  ML4 

ML  ft . 

B0%ofJVlU  and  ML3SPS 
.  • : . .  i^S^.ol  Uucugh  MLb 


Mostgsnertc  practices  are  also 
addressed. 


Dd:*  SCATi^J  C  i}(  tna  Jiteft  a 


Organizations  Using  TSP 


Microsoft  softtek 

OQAOjC 

TOSHIBA  FUJIfILM 


Intuit 

M 


TPiyr 


^Sun 


«  lA  me  . 


•i^r  o»  <1*  Lt» 


V;/C  *iitr4>«M»c^«»^tAMC  >» 


tM- W 


NAVAiR  AV-8B  TSP/CMMI  Experience 


AV-8B  is  a  NAVAIR  System 
Support  Activity. 

They  integrate  new  features 
into  the  Marine  Harrier 
aircraft 

They  used  TSP  to  reduce 
the  time  to  go  from  CMMI 
Level  1  to  CMMI  Level  4. 

SEI  Average 

AV-8B 


u  t  •u.voctM^a'te^ 


Reliable  Products 


.  An  ahaJysls.  Qf  20  projects,  in.  1.3. 

. organizations,  showed  TSP,  teams. 

averaged  0.06; defects  per 
thousand  lines;  of  new  or  modified 
code.: . 


Approximately  M  /3  of  these 
projects  were  defect-free. 


These  results  are  substantiaiiy 
better  than  those  achieved  in  high 
maturity  organizations 

Source:  CMU/SEI-2003-TR-014 


Productivity  Improvement 


f  rorin  data  orrover~4Q  TSP  teams’ Intuit  has  found  that  ””r*: 
I  •  post  code-complete  effort  is  8%  inkead  of  33%  of  the  project 


I  •:  for  TSP 

projects 

standarc 

j  test  times  are  cu 

t  from  4  r 

nonths  to 

:  1  month 

f . ;■  or  less'. 

Organizations  using  TSP  report  productivity  gains  of  30%  to  80% 
Resulting  in  iov/er  costs  or  more  functionality  in  delivered  software. 
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Individual  Transition 


w 

w 

P 

Contact 

Awareness 

Understanding 

Trial  Use 

Adoption 

•Conversation 

•Conferences 

•JIT  Training  Focused 

•Packaged  proven 

•  Project  Based 

•Website 

•Books 

on  the  projects  and 

whole  product 

Rollout  Strategy 

units  implementing  the 

Launch  Process 

•Organizational 

Commitment 

•Article 

•Articles 

processes(two  weeks) 

•Supported  by  a 

•Training 

•Three  levels  of 
training 

•Executive 

“COACH” 

•  Organizational 

•Instrumented 

Support  (EPG) 

•Team  Leader 

•Implements  the 

•Practitioner 

Processed  learned 
in  the  Learning 

Laboratory  on  the 

•Advanced  Learning 

actual  project 

Laboratory 

•Coach  reinforces 
discipline 
throughout  the 
project 
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Advanced  Learning  Laboratory 


Training  ++  Behavioral  modification 

Process  Simulation  Challenge  current  beliefs 

Individual  Instrumentation  Change  Behavior 

Immersion  Therapy  Change  Behavior  generates  new  results 

Self  Discovery 
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Process  Simulation 
Executing  the  Processes 


Results  from  executing  the  Process 

Product-Process-Planning  Data 
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Belief  Systems  and  Behavior 


Belief  drives  behavior 


How  to  change  a  belief? 


•  Behavior 


•  Change  the 
behavior 


Show  results  inconsistent  with  the  belief 
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My  Beliefs-My  Data-  My  Journey 


^Failure  Cost  of  Quality 

_  H  X 

Failure  Cost  of  Quality 
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EFFORT 


Consciousness  Model  and  Bandura  Social  Learning 
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Bayesian  Belief  networks 


Bayesian  Inference  Model:  Allow  the 
use  of  prior  knowledge. 

\-e{P(h\Q  be  a  degree  of  belief  in  h 
given  current  state  of  information 

New  evidence  e  is  presented. 

Update  using  Bayes’s  Theorem: 


P{h\e,^) 


P{h\^)P{e\h,^) 

Pie  I 
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Predicting  Behavior  based  on  Beliefs 


The  Technology  Acceptance  Model  is  an  information 
systems  theory  that  models  how  users  come  to  accept 
and  use  a  technology 


Simplified  Acceptance  Model  based  on  Beliefs 

Repeated  for  Contact,  Awareness,  Understanding,  Trial  use  and 
Institutionalization 


Bagozzi,  R.  P.,  Davis,  F.  D.,  &  Warshaw,  P.  R.  (1992). 

Development  and  test  of  a  theory  of  technological 
learning  and  usage.  Human  Relations,  45(7),  660-686. 
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Concept  of  a  BBN  Model 


Contact 

Awareness 

Understand! 

Trial  Use 

Institution  - 

ng 

alization 
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Using  BBN  Model  to  Predict  Future 


Contact 


Awareness  Understand! 

ng 


Trial  Use 


1 

^  Benefi 

For  a  particular  client  at  a  given  point  in 
the  adoption  curve,  knowledge  of  any  of 
the  past  or  present  scores  can  be  used  to 
predict  the  future  scores! 


Continu 
e 


Institution 

alization 


Changing  Behavior 
©2009  Carnegie  Mellon  University 


Using  BBN  Model  to  Explain  Past 


Contact 

Awareness 

Understand! 

Trial  Use 

Institution  - 

ng 

alization 

\ 

For  a  particular  client  at  aijlveri  point  In  ' 
the  adoption  curve,  knowledge  of  a  recent 
score  can  be  used  to  explain  what  the 
historical,  unknown  scores  most  likely 
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Transition  Survey 


Awareness: 

Activity:  ExecutiveSenninar/Teann  leadtraining 


PSP  will  benefit  me/my  organization: 

No 

0% . 25  . 50. . 75, 


PSP/TSP  wili  work  for  me/ my  organization 
No 

0% . 25  . 50. . 75, 


(ForPartners]  What  percentage  of  clients  continue  on  to  understanding 
0% . 25  . 50. . 75, 


Comments: 


Software  Engineering  Institute 
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Yes 

.100% 

Yes 

.100% 

.100% 


Mellon 


Changing  Behavior 


25 


©2009  Carnegie  Mellon  University 


Changing  Benefit  Profiie 


Software  Engineering 


This  distribution  of  the  Benefit  score  is 
noticeabiy  moving  up  across  the  adoption 

phases 
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Changing  Work  Profile 


r 


A 


Software  Engineering 


This  distribution  of  the  Work  score  is  noticeabiy 
moving  up  across  the  adoption  phases 
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Changing  Continue  Profiie 


Software  Engineering 


This  distribution  of  the  Continue  score  is 
noticeabiy  moving  up  across  the  adoption 

phases 
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Overall  Trend  of  Average  Responses 


Benefit 

•Work 

Continue 
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Some  Initial  Linear  Models 


Contact-Continue-Score  =  4.3  +  0.85  *  Contact- 

Work-Score 

(Adj-Rsquare  =  48%) 


Understand-Br  Although  we  prefer  adjusted  Rsquare 

Aware  values  in  the  80%+  range,  these  single 

factor  prediction  modeis  show  promise. 


(Adj  Remember,  Adj-Rsquare  is  the  amount  of 
behavior  of  the  outcome  explained  by  the 

modeling  factor 


Questions? 
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Backup  and  Reference  slides  follow 
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Software  Industry  Project  Performance 


iliUihl 


2000 

2002 

2004 

2006 

2008 

28% 

34% 

29% 

35% 

32% 

23% 

15% 

18% 

19% 

24% 

49% 

51% 

53% 

46% 

44% 

■  Succeeded  ■  Failed  ■  Challenged 


to  2008  Project  Resolution 


Successful  projects  delivered  on  time, 
on  budget,  with  required  features  and 
functions. 

Challenged  projects  were  late,  over 
budget,  and/or  failed  to  deliver  all  of 
the  required  features  and  functions. 

Failed  projects  were  cancelled  prior  to 
completion  or  delivered  and  never 
used. 


Source:  Standish  group  2009  Chaos 
report. 
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Software  Industry  Quality  Performance 


The  software  industry  is  the  only  modern 
high-tech  industry  that  ignores  quality  until 
test. 


Most  software  defects  are  found  in  or  after 
test  when  defect  removal  costs  are  the 
highest  and  the  methods  are  the  least 
effective. 


This  strategy  results  in  defective  products 
and  unnecessary  rework  that  inflates 
development  costs  by  30%  to  40%  or 
more. 


This  strategy  is  also  a  principal  cause  of 
unexpected  delays,  system  failures,  and 
software  security  vulnerabilities. 
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Competitive  Advantage 


As  competition  in  the  software  industry 
increases,  organizations  seek: 

•  lower  development  cost 

•  shorter  schedules 

•  more  features  per  release 

•  predictable  plans 

•  improved  product  quality 

•  fewer  customer  reported  defects 

•  reduced  staff  turnover 


Team  Software  Process  supports  these 
objectives. 
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Reliable  Estimates 


From  a  study  published  in  2000 

•  fifteen  projects  in  four 
organizations 

.  CMM  ML1,  ML2,  MLS,  and  MLS 

•  TSP  improved  effort  and  schedule 
predictability  at  all  maturity  levels 


Effort  (Cost)  Performance 

Study  baseline 

+17%  to  +85% 

TSP 

-25%  to  +25% 

Schedule  Performance 

Study  baseline 

+27%  to +112% 

TSP 

-8%  to  +20% 

Source:  CMU/SEI-TR-2000-015 


Effort  Deviation  Range 


Schedule  Deviation  Range 


120% 

100% 

80% 

s 

lii  60% 

g  40% 
0) 

Q-  20% 
0% 
-20% 
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Reliable  Products 


An  analysis  of  20  projects  in  13 
organizations  showed  TSP  teams 
averaged  0.06  defects  per 
thousand  lines  of  new  or  modified 
code. 


Approximately  1/3  of  these 
projects  were  defect-free. 


These  results  are  substantially 
better  than  those  achieved  in  high 
maturity  organizations. 


Source:  CMU/SEI-2003-TR-014 
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Defects/KLOC 


Level  1 


Level  2 


Level  3 


Level  4 


Level  5  TSP 
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Reduced  Rework 


TSP  System  Test  Performance  Range  and  Average 


60% 

50% 

40% 

30% 

20% 

10% 

0% 


Range  of  a 

Typical  Project 

1 

1 

1 

1 

◄ 

► 

\ 

1 

1 

1 

System  Test  Effort 
%  of  Total 


System  Test 
Schedule  %  of 
Total 


Failure  COQ 


Source:  CMU/SEI-TR-2003-014 
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Productivity  Improvement 


From  data  on  over  40  TSP  teams,  Intuit  has  found  that 

•  post  code-complete  effort  is  8%  instead  of  33%  of  the  project 

•  for  TSP  projects,  standard  test  times  are  cut  from  4  months  to  1  month 
or  less. 


Non-TSP 


Development 


Test 


TSP 


Development 


Test 


Organizations  using  TSP  report  productivity  gains  of  30%  to  80% 
resuiting  in  lower  costs  or  more  functionaiity  in  deiivered  software. 
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A  Process  for  Managers  and  Developers 


"'It  nice  to  be  associated  with  a  pro¬ 
ject  that  had  few  defects.  " 

'■'■77?^  system  test  e}igineers  became  con¬ 
vinced  that  TSF  was  woj'thwhile  when  they' 
realized  that  tke}>  ivere  going  from  track¬ 
ing  down  sojhvare  bugs  in  the  lab  to  Just 
confirming  functionalityK  Our  first  pro¬ 
ject:  certified  with  ten  times  increase  in 
quality^  with  significant  drop  in  cost  to  de¬ 
velop.  FoIIow-on  project:  certified  with 
NO  software  defects  delivered  to  system 
test  or  customer. '' 

''One  of  my  first  projects  as  an  embedded 
systems  programmer  finished  on  the  day 
we  planned  to  finish  six.  months  earlier.  I 
attribute  the  success  to  planning  at  a  bet¬ 
ter  gi'anularity  and  making  full  use  of  the 
earned  value  hacMng.  The  day  ii'e  got 
1 00%  earned  value  was  the  da\^  we 
planned  to  get  100%  value,  and  we  as  a 
team  celebrated  like  we  had  a  bas¬ 
ketball  game. '' 


"My  first  TSF-based  team  lecently  fin¬ 
ished  their  system  test  They  had  three  sys¬ 
tem  test  defects  in  7400  lines  of  code.  No 
dfiects  were  code-  or  design-related;  they 
were  either  install  or  documentation — 
each  of  which  took  about  five  minutes  to 
fix.  System  test  took  less  than  five  percent 
of  the  overall  project  effort.  ” 

"Multiple projects  in  our  organization 
have  been  able  to  keep  within  their  time 
schedules  (-\-/-  three  weeks)  over  a  six- 
month  span.  This  is  something  we  [had] 
not  been  able  to  accomplish  in  the  past. 
This  is  one  of  the  reasons  that  manage¬ 
ment  is  happy  v4th  the  TSF  process.  " 

"Our  schedule  f'eliability  is  now  +/-  ten 
percent  from  -501^700  percent  and  our 
defect  density^  at  the  team  level  has  been 
reduced  by  over  50  percent '' 

"Measuring  pj'ogress  helps  generate  pro¬ 
gress.  ” 


"...[TSF  is  a]  transparent  project  man¬ 
agement  paradigm — everybody  has  a 
common  understanding  of  the  plan  and 
e)'eryone  knows  what  is  going  on  in  the 
project  and  where  vi'e  are  in  the  project  at 
any  time. " 

"Our plans  are  much  7nore  detailed  and 
all  the  involved  de\^eIopers  understand 
them.  As  a  consequence,  we  deliver  what 
we  planned,  on  time. '' 

"PSF  really  sells  you  on  the  idea  about 
finding  defects  early  in  the  process.  It 
really  does  7nake  a  dijference  at  the  end. 
Wb  thought  it  Timri  t  going  to  work.  But 
we  all  becafne  converts.  In  doing  the 
wor'k  you  are  producing  valuable  data 
along  th  e  way.  We  impr  oved  productiv¬ 
ity^...  improved  it  greatly.  I  worried  be¬ 
cause  I  have  seen  too  many  people  more 
interested  in  the  process  than  in  the  prod¬ 
uct  You  ar  e  ft nishing  small er  products  at 
7nore  regular  intervals.  ” 


Source:  CMU/SEI-TR-2003-014 
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TSP  Implements  CMMI  -1 


a. 


(n 


w 

4- 

§ 


a. 


CMAAI  Process  Categories 


□  Unrated 

■  Not  Addressed 

□  Partially  Addressed 

□  Supported 

□  Uirectly  Addressed 


Unrated  -  out  of  scope 
for  TSP. 


Not  addressed  - 

project  practice  that 
TSP  does  not  cover. 

Partially  addressed  - 

project  practices  that 
TSP  addresses  with 
some  weakness  of 
omission 

Supported  - 

organizational 
practices  that  TSP 
supports. 

Directly  Addressed  - 

TSP  practices  meet 
the  intent  of  the  CMMI 
specific  practice  (SP) 
without  significant 
reservations. 


Based  on  a  SCAMPI  C  of  the  latest  version  of  TSP 
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TSP  Implements  CMMI  -2 


An  organization  using  TSP  has  directly 
addressed  or  implemented  most 
specific  practices  (SP). 

.  85%ofSPsatML2 

.  78%  of  SPs  at  MLS 

.  54%  of  SPs  at  ML4 

.  25%  of  SPs  at  MLS 

.  80%  of  ML2  and  MLS  SPs 

•  75%  of  SPs  through  MLS 


CAIUMI  /Maturity  Level 


Most  generic  practices  are  also 
addressed. 


Based  on  a  SCAMPI  C  of  the  latest  version  of  TSP 
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NAVAIR  AV-8B  TSP/CMMI  Experience 


AV-8B  is  a  NAVAIR  System 
Support  Activity. 

They  integrate  new  features 
into  the  Marine  Harrier 
aircraft. 

They  used  TSP  to  reduce 
the  time  to  go  from  CM  Ml 
Level  1  to  CMMI  Level  4. 


SEI  Average 

6  Years 

AV-8B 

2.5  Years 
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Organizations  Using  TSP 


H^CrOSOft  Softtek'  ' 

OPACLe 

TOSHIBA  FUJSfILM 

Leading  Innovation  >» 


Mlllt 


Adobe 


Vicarious 

Visions^ 


microsystems 


Advanced  Information  Services,  Inc. 

Centro  De  Investigacion  En  Matamaticas 
Chinasoft  International,  Inc. 
computing  Technologies,  Inc. 

Davis  Systems 

DEK  International  GmbH 

Delaware  Software,  S.A.  de  C.V. 

Delivery  Excellence 

Grupo  Empresarial  Eisei,  S.A.  de  C.V. 

Herbert  Consulting 

Hitachi  Software  Engineering  Co.,  Ltd. 

Idea  Entity  Corp. 

InnerWorkings,  Inc. 

Institute  Tecnologico  y  de  Estudios  Superiores  de 
Monterrey 

It  EraS,A,  de  C,.V. 

Kernel  Technologies  Group,  S.A.  de  CV 


Knowldege  Partner  QR  Pvt.  Ltd. 

Kyushu  Institute  of  Technology 
L.  G.  Electronics 
Log  i  Care 
Motive,  LLC 

National  Aeronautics  &  Space  Administration 
Next  Process  Institute  Ltd. 

Praxis  High  Integrity  Systems 
Process  &  Project  Health  Services 
Procesix 

PS&J  Consulting  -  Software  Six  Sigma 
QuarkSoft 

Sandia  National  Laboratories 

Science  Applications  International  Corporation 
(SAIC) 

Siemens  AG 


SILAC  Ingenieria  de  Software  S.A.  de  C.V. 
SKIZCorp  Technology 

Software  Engineering  Competence  Center  (SECC) 
Software  Park  Thailand 
STPP,  Inc. 

TOWA  INTEGRADADORA  S.A.  de  C.V. 

TRX 

Universidad  Autonoma  De  Zacatecas 
Universidad  de  Monterrey 
Universidad  Regiomotana  A.C. 

University  of  Aizu 

U.S.  Air  Force  (CRSIP/STSC) 

U.S.  Census  Bureau 

U.S.  Navy  Air  Systems  Command  (NAVAIR) 

U.S.  Naval  Oceanographic  Office  (NAVO) 
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Topics 


Introduction 
TSP  concepts 

•  Self-directed  teams  and  coaching 

•  Personal  Software  Process 

•  Process  and  measurement  framework 

•  Comprehensive  quality  management 

Team  management  with  TSP 

User  experience 
Getting  Started 
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Key  Features  -1 


Unlike  many  other  software  development  methods  TSP  a  uses  self-directed 
team  management  style... the  team  owns  the  plan. 


TSP  has  an  operationally  defined  process  that  is  also  owned  by  the  team. 


The  process  is  supported  by  an  integrated  measurement  framework  to  help  the 
team  track  their  work  and  improve  their  estimating  abilities. 


TSP  emphasizes  quality  with  comprehensive  quality  management  practices. 

•  build  the  right  product  the  right  way  to  avoid  rework 

•  put  quality  product  into  test  instead  of  trying  to  test-in  quality 
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Key  Features  -2 


Complete  engineering  process  -  system  requirements  through 
acceptance  test. 


Scalable  -  small  to  large  organizational  settings  and  projects. 


Tailorable  -  TSP  is  tailored  or  is  adapted  to  support  existing 
processes. 


Provides  immediate  and  measurable  benefits  on  first  use. 


Role  specific  training,  documented  process,  and  tools. 
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Topics 


Introduction 
TSP  Concepts 

•  Self-directed  teams  and  coaching 

•  Personal  Software  Process 

•  Process  and  measurement  framework 

•  Comprehensive  quality  management 

Team  management  with  TSP 

User  experience 
Getting  Started 
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Management  Styles 


The  principal  management  styles  have  been: 


Body  Management 

People  as  oxen  that  must 
be  driven,  directed,  and 
motivated  through  fear. 


Frederick  Taylor 


Task  Management 

People  as  machines. 
Management  knows  the 
best  way  to  get  the  work 
done.  The  workers  follow 


Peter  Drucker 


Knowledge  management 

People  as  individuals.  The 
knowledge  worker  knows  the 
best  way  to  get  the  work  done. 
Management  motivates,  leads, 
and  coaches. 
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Knowledge  Work 


“The  key  rule  in  managing  knowledge  work  is 
this:  managers  can’t  manage  it,  the  workers  must 
manage  themselves.” 

Software  development  is  knowledge  work. 

To  manage  software  work,  developers  must 

•  be  motivated 

•  make  accurate  plans 

•  negotiate  commitments 

•  track  their  plans 

•  manage  quality 

How  is  this  accomplished? 


♦ 


Watts  Humphrey, 
creator  of  TSP 
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TSP  Self-directed  Team  Management  Style 


Traditional  team 

The  leader  plans,  directs,  and 
tracks  the  team’s  work. 


Self-directed  team 

The  team  members  participate  in 
planning,  managing,  and  tracking  their 
own  work. 
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Sharing  the  Team  Management 
Responsibilities 


Self-directed  team  roles 

Eight  pre-defined  roles  distribute  traditional 
project  management  responsibilities  across  the 
team. 

All  team  members  have  traditional  roles,  e.g. 
developer,  tester,  etc. 


Project  Management  Roles 


Planning  manager  -  responsible  for  tracking  the  plan. 

Quality  manager  -  responsible  for  tracking  the  quality  plan. 

Process  manager  -  responsible  for  ensuring  process 
discipline  and  for  process  improvement. 

Support  manager  -  responsible  for  ensuring  that  support 
needs  are  met  and  for  configuration  management. 


Technical  Roles 


Customer  interface  manager  -  responsible  for  the  interface 
to  the  customer  or  customer  representative. 

Design  manager  -  responsible  for  the  design  practices  and 
quality. 

Implementation  manager  -  responsible  for  implementation 
practices  and  quality. 

Test  manager  -  responsible  for  test  practices  and  quality. 
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The  Team  Leader’s  Role 


The  team  leader  does  not  typically  take  one  of  the  eight  team  member 
roles. 

The  team  leader’s  job  on  a  TSP  team  is  to 

•  guide  and  motivate  the  team  in  doing  its  work 

•  take  the  time  to  reach  full  consensus  on  all  important  issues 

•  ensure  that  the  team  establishes  high  standards  for  the  work 

•  provide  management  support  to  the  team 

•  support  the  team  with  management 

•  protect  the  team  so  that  it  can  concentrate  on  the  project 
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The  TSP  Coaching  Role 


The  coach 

•  trains  and  facilitates  the  adoption  of  TSP 

•  works  with  the  team  leader  to  build  the  team 

•  observer  that  guides  the  team 


Team  Leader  vs.  Coach 

The  team  leader’s  job  is  to  use  the 
team  to  build  the  product. 

The  coaches  job  is  to  use  the  project 
to  build  the  team. 

Tiger  Woods  and  his  coach  Hank  Haney. 
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The  Impact  of  Self-Directed  Teams 


A  self-directed  team 

•  builds  its  own  plans,  negotiating  trade-offs  with  management. 

•  owns  its  process  and  is  committed  to  following  it. 

•  measures  and  tracks  its  own  work. 

•  knows  precisely  where  it  stands. 

Because  of  this  the  team  members  are  highly  motivated  to  help  each 
other  meet  their  commitments  and  achieve  their  best  performance. 
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Topics 


Introduction 
TSP  Concepts 

•  Self-directed  teams  and  coaching 

•  Personal  Software  Process 

•  Integrated  process  and  measurement  framework 

•  Comprehensive  quality  management 

Team  management  with  TSP 

User  experience 
Getting  Started 
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Learning  to  Develop  Software 


In  universities, 

•  the  emphasis  is  on  technical  knowledge  and  individual  performance. 

•  evaluation  emphasizes  code  that  runs,  not  how  the  student  got  there. 

•  the  prevailing  ethic  is  to  code  as  quickly  and  fix  the  problems  in  test. 

In  industry,  team-working  skills  are  also  needed. 

TSP  uses  the  Personal  Software  Process  to  build  these  skills. 

•  planning  and  tracking  the  work 

•  measuring  and  managing  quality 

•  anticipating  and  correcting  problems 
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PSP  Learning  Stages 


Introduces  quality 
management  and  design 


Introduces  estimating  and 
planning 


Introduces  process  discipline 
and  measurement 


Developers  write  one  or  more  programs  at  each  PSP  level 
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PSP  Estimating  Accuracy 


Majority  are  under-estimating 


Balance  of  over-estimates  and  under¬ 
estimates 


Much  tighter  balance  around  zero 
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Compile  and  Test  Defects  -  from  PSP  Training 
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Mean  Minutes  Spent  Per  LOC 


PSP  Design  Time  Resuits 


Time  Invested  Per  (New  and  Changed)  Line  of  Code 


Design 

-• -  Code 

■o -  Compile 

Test 


298  developers 


^  Software  Engineering  Institute  CarnegieMellon 


Changing  Behavior 
©2009  Carnegie  Meiion  University 


61 


Topics 


Introduction 
TSP  Concepts 

•  Self-directed  teams  and  coaching 

•  Personal  Software  Process 

•  Process  and  measurement  framework 

•  Comprehensive  quality  management 

Team  management  with  TSP 

User  experience 
Getting  Started 
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TSP  Operational  Processes  and  Measures 


TSP  is  defined  operationally. 

•  The  processes  provide  guidance  without  being  too  detailed  or  inflexible. 

•  They  are  easily  tailored  to  fit  existing  organizational  processes. 

•  The  measurement  definitions  are  precise  but  also  extensible. 

Benefits 

•  Allows  self-directed  teams  to  own  their  processes. 

•  Instills  process  discipline  rather  than  enforcing  process 
institutionalization  with  auditing  methods. 
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TSP  Process  Structure 


The  TSP  process  elements  can  be 
organized  into  whatever  process 
structure  makes  the  most  business 
and  technical  sense. 


The  phases  can  be  implemented 
iteratively  in  small  cycles,  in  a  spiral 
with  increasing  cycle  content,  or 
sequentially  as  in  a  waterfall. 


TSP  projects  can  start  on  any 
phase  or  any  cycle. 


Each  cycle  starts  with  a  launch  or 
re-launch  and  ends  with  a 
postmortem. 


Business 

and 

technical 

goals 


Estimates,  plans, 
process,  commitment 


Lessons,  new 
goals,  new 
requirements, 
new  risk,  etc. 


Development 
phase 
or  cycle 


Phase  or  cycle 
Postmortem 


Work  products, 
status,  metrics, 
results 


Project 
Postmortem 
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The  TSP  Launch  Process 


The  most  important  outcome  is  a 
committed  team. 
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The  TSP  Development  Process 


Produce 

Requirements 

Specifications 


The  TSP  process  elements  are 
adapted  to  the  organization’s  process. 
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Measurement  Framework 


Schedule 


Effort 


Size 


Quality 
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Four  base  measures 


Apply  to  all  processes 
and  products 

Estimates  made  during 
planning 

Directly  measured  by 
team  members  while 
working 
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Schedule 


Schedule  is  the  most  commonly  used  project  measure. 
Schedule  accuracy  depends  on  granularity. 


TSP  schedule  granularity  is  in  hours,  not  days,  weeks,  or 


months. 

_ LJ _ \ _ = _ I _ : _ 0 _ u _ I _ = _ I _ Lu _ I _ u _ I _ I _ I _ ^ _ I _ _ I _ u _ I _ z _ I _ ^ _ \ _ : _ 

TSP  Task  Planning  Template  -  Form  TASK  Total  Plan  Hours  Total  Actus 

Name  Prasad  Perini _  _ 31 S  .9  _ 

Team  PSP  Ghost _  Reminder: 

Date  2/3/2004 _  Estimated  Hours  can  be  entered  manually  -  OR  -  calculated  based  on  Estimated  Si: 

_  If  Size  and  Rate  are  present,  this  field  will  be  recalculated  when  you  Update  Task 


Cycle  ^ 


Assembly 

Phase 

Generate 
Task  List 

Update  Task 
and  Schedule 

Resources 

Estimated  Size 

Size  Measure 

Rate  (per  Hr.) 

Estimated  Houi 

Engrs 

Plan  Hours 

Plan  Date 

Plan  Week 

Actual  Hours 

Actual  Date 

Task 

Main  Form 

DLDINSP 

Main  Form  DLD  Inspection 

SA,  PP 

300 

LOC 

200.0 

1.5 

1.0 

1.5 

3/1 0/2003 

15 

5.0 

3/7/2003 

Main  Form 

CODEINSP 

Main  Form  Code  Inspection 

SA.PP 

300 

LOC 

200.0 

1.5 

1.0 

1.5 

3/1 0/2003 

15 

4.8 

3/1 0/2003 

Filter  Object 

CODEINSP 

Filter  Object  Code  Inspection 

SA,  PP 

300 

LOC 

200.0 

1.5 

1.0 

1.5 

3/1 0/2003 

15 

3.2 

1  /22/2003 

Task  Panel  Control 

DLDINSP 

Task  Panel  Control  DLD  Inspection 

NK,PP 

250 

LOC 

200.0 

1.3 

1.0 

1.3 

3/1 0/2003 

15 

0.0 

3/7/2003 

Task  Panel  Control 

CODEINSP 

Task  Panel  Control  Code  Inspection 

NK,PP 

250 

LOC 

200.0 

1.3 

1.0 

1.3 

3/1 0/2003 

15 

0.0 

3/1 0/2003 

ProfileUserList.aspx  DLDINSP 

ProfileUserList.aspx  DLD  Inspection 

PP,  VY 

1010 

LOC 

200.0 

5.1 

1.0 

5.1 

3/1 7/2003 

16 

2.0 

2/4/2003 

ProfileUserList.aspx 

CODEINSP 

ProfileUserList.aspx  Code  Inspection 

PP,  VY 

1010 

LOC 

200.0 

5.1 

1.0 

5.1 

3/1 7/2003 

16 

4.4 

2/27/2003 
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Time 


Time  is  a  measure  of  time  on  task. 


The  TSP  time  measure  is  task  hours, 
i.e.  the  time  spent  on  a  project  task, 
minus  interruption  time. 


TSP  team  members  record  their  time 
as  they  work,  not  at  the  end  of  the 
day,  week,  or  month. 


Software  Engineering  Institute 


TSP  Time  Recording  Log  -  Form  LOGT 

Name  Prasad  Perini _ 

Team  PSP  Ghost 


Date  2/3/2004 


Cycle 

Hours 

321.2 

Assembly 

Phase 

Task 

Date 

Start 

Int. 

Stop 

Delta 

OEM-ChangeRPLAN 

OEM-ChangeRequest-7  PLAN 

03/1 3/03 

15:45:10 

16:22:43 

37.8 

OEM-ChangeR  HLD 

OEM-ChangeRequest-7  HLD 

03/1 3/03 

18:53:08 

17:30:40 

37.5 

OEM-ChangeR  DLD 

OEM-ChangeRequest-7  DLD 

03/1 3/03 

17:30:49 

18:02:59 

32.2 

OEM-ChangeR  DLD 

OEM-ChangeRequest-7  DLD 

03/1 3/03 

18:55:20 

19:54:35 

59.3 

OEM-ChangeR  DLDR 

OEM-ChangeRequest-7  DLDR 

03/1 4/03 

10:00:43 

10:31:59 

31 .3 

OEM-ChangeR  DLDINSP 

OEM-ChangeRequest-7  DLDINSP 

03/1 7/03 

14:37:38 

15:13:58 

38.3 

OEM-ChangeR  DLD 

1  OEM-ChangeRequest-7  DLD 

03/1 7/03 

15:46:18 

16:00:51 

14.8 

OEM-ChangeR  DLD 

OEM-ChangeRequest-7  DLD 

03/1 7/03 

16:11:56 

16:33:34 

21 .6 

OEM-ChangeR  DLDR 

OEM-ChangeRequest-7  DLDR 

03/1 7/03 

16:46:49 

17:04:20 

17.5 

OEM-ChangeR  CODE 

OEM-ChangeRequest-7  CODE 

03/1 7/03 

17:45:47 

18:47:23 

61 .6 

OEM-ChangeR  CODE 

OEM-ChangeRequest-7  CODE 

03/1 7/03 

18:50:51 

19:01:18 

10.5 

OEM-ChangeR  CODE 

1  OEM-ChangeRequest-7  CODE 

03/1 8/03 

09:38:54 

10:10:35 

31 .7 

OEM-ChangeR  CR 

OEM-ChangeRequest-7  CR 

03/1 8/03 

11:50:48 

12:04:33 

13.8 

OEM-ChangeR  CR 

OEM-ChanqeRequest-7  CR 

03/1 8/03 

12:53:56 

13:29:14 

35.3 
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Size  is  a  measure  of  the  magnitude  of  the 
deliverable,  e.g.  lines  of  code  or  function  points, 
pages. 


TSP  size  measures  are  selected  based  on  their 
correlation  with  time. 


f 


TSP  also  uses  size  data  to 

•  normalize  other  measures 

•  track  progress 


TSP  Size  Summary  -  Form  SUMS 

Name  Prasad  Perini 

Team  PSP  Ghost 

Date  2/3/2004 

cycle 

Actual  Size 

Q 

Assembly,  Sub-Assembly, 
or  Part  Name 

03 

O 

>■ 

n. 

E 

0 

(0 

g 

Parent  Assembly  Name 

0 

5 

Size  Measure 

Base 

Deleted 

Modified 

■c 

QJ 

T3 

Reused 

New  and  Changed 

Total 

25 

DeliveryOEMPartValidate-Flle; 

A 

OEM  MOO  Integration  RSM 

PP 

LOO 

0 

0 

0 

489 

0 

489  489 

26 

27 

28 

29 

30 

DeliveryOEMPartLIstCSQL) 

A 

OEM  MOO  Integration  RSM 

PP 

LOO 

0 

0 

0 

613 

0 

61 3  61 3 

178  178 

153  153 

4  4  •' 

At3pDataExchangeCreate(SQI 

A 

'OEM  MOO  Integration  RSM 

PP 

LOO 

0 

0 

0 

178 

0 

AppDataExchangeGet(SQL) 

A 

OEM  MOO  Integration  RSM 

PP 

LOO 

0 

0 

0 

153 

0 

OEM  MOO  Integration  RSM 

^  _J 

[system 

|nk 

Text  Pages 

0 

0 

0 

4 

0 

Build  Doc  for  OEM  MOO  Teanf 

i  A  I 

:OEM  MOO  Integration  RSM 

NK 

Text  Pages 

0 

0 

0 

0 

0 
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Defects 


Defects  are  the  measure  of  quality  in  the  TSP. 


Any  change  to  an  interim  or  final  work  product,  made 
to  ensure  proper  design,  implementation,  test,  use, 
or  maintenance,  is  a  defect  in  the  TSP. 


Defects  are  logged  as  they 
are  found  and  fixed. 


Defect  tracking  takes  place 
throughout  the  process. 


TSP  D€ 

Name 

Team 

jfect  Recording  Log  - 1 

Prasad  Perini 

=orm  LOG 

Date 

D 

2/3/2004 

PSP  Ghost 

Cycle 

Date 

Num 

Type 

Assembly 

Injected 

Removed 

Fix 

Time 

Fix 

Ref. 

Description 

1  n  6/2003 

66 

20 

OEM  User  Groups 

CODE 

CR 

5.0 

Missing between  parameters 

1  /1 6/2003 

67 

70 

OEM  User  Groups 

CODE 

CR 

5.0 

Permissions  don't  match  for  objects  and  its  attribul 

1  /23/2003 

68 

70 

OEM  User  Groups 

DLD 

CODEINSP 

5.0 

SRFile,  SRProperty  objects  need  create  permissioi 

1  /23/2003 

69 

70 

OEM  User  Groups 

DLD 

CODEINSP 

10.0 

Permissions  don't  match  for  objects  and  its  attribul 

1  /23/2003 

70 

70 

OEM  User  Groups 

CODE 

CODEINSP 

2.0 

21 1  -21 2  VWong  Sproc  (iGrpApp  should  be  iCode) 

1  /24/2003 

71 

70 

OEM  User  Groups 

CODE 

UT 

25.0 

VWong  Database  Name  for  User  Account  Object 

1  /24/2003 

72 

70 

OEM  User  Groups 

DLD 

UT 

3.0 

'Extra  Attribute  name  in  User  Account  Object  Attribi 

1  /24/2003 

73 

90 

AppDataExchangeG 

DLD 

DLDR 

1.0 

Granted  permissions  to  OEMUsers  instead  of  Phof 

1  /24/2003 

74 

40 

AppDataExchangeO 

DLD 

DLDR 

5.0 

Step  names  in  Logic  don't  match  with  error  table 

1  /24/2003 

75 

40 

AppDataExchangeG 

DLD 

DLDR 

1.0 

Change  record  to  IsActive  in  step  2 

1  /24/2003 

76 

70 

AppDataExchangeG 

DLD 

DLDR 

1.0 

- 1 

Column  names  were  not  specified  in  step  4 
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What  the  Base  Measures  Provide 


Management  measures  derived  from  the  base  measures  are  used  by 
the  team  to  manage  the  project  and  manage  quality. 


Project  management  measures:  earned  value,  productivity  , 
estimation  accuracy,  estimation  size  and  effort  prediction  intervals,  cost 
performance  index,  time  in  phase  distributions,  ... 


Quality  management  measures;  defects  injected  and  removed  in 
each  process  phase,  defect  density,  defect  injection  and  removal  rates, 
process  yield,  phase  yield,  review  and  inspection  rates,  cost  of  quality, 
percent  defect  free,  quality  profiles,  quality  profile  index,  ... 
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Topics 


Introduction 
TSP  Concepts 

•  Self-directed  teams  and  coaching 

•  Personal  Software  Process 

•  Process  and  measurement  framework 

•  Comprehensive  quality  management 

Team  management  with  TSP 

User  experience 
Getting  Started 
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Testing  Coverage 


Overload 


Safe  and  secure 
region  =  tested 
(shaded  green) 

Unsafe  and  insecure 
region  =  untested 
(shaded  red) 
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Put  a  Quality  Product  into  Test 


IBM’s  Dr.  Harlan  Mills  asked:  “How  do 
you  know  that  you’ve  found  the  last 
defect  in  system  test?” 


“You  never  find  the  first  one. 


If  you  want  a  quality  product  out  of 
test,  you  must  put  a  quality  product 
into  test. 


How  do  you  put  a  quality  product  into 
test? 


Quality  Management! 
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TSP  Quality  Management  Practices  -1 


Planning  for  quality 

•  TSP  quality  planning  estimates  the  number  of  defects  injected  and 
removed  at  each  phase  based  on  historical  injection  rates  and  phase 
yields. 

•  Removal  rates,  review  rates,  phase  time  ratios,  defect  densities,  and 
other  quality  indicators  are  then  calculated  by  the  tools. 

Measuring  and  tracking  quality 

•  Developers  track  every  defect  found  and  fixed. 

•  Quality  is  reviewed  weekly  by  the  quality  manager  and  the  team. 
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TSP  Quality  Management  Practices  -2 


Defect  removal  filters 


•  Every  activity  that  finds  and  removes  defects  can  be  thought  of  as  a 
defect  removal  filter,  e.g.  reviews,  inspections,  compilers,  static 
analyzers,  etc. 

•  TSP  has  many  such  filters. 

Ca  ptu  re/Reca  ptu  re 

•  TSP  uses  capture/recapture  to  estimate  the  defects  missed  in 
inspections. 

Defect  prevention 

•  Every  defect  found  in  system  test  or  later  is  analyzed  to  prevent  future 
escapes. 

•  Every  defective  module  is  re-inspected. 
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Quality  and  the  Team 


High  quality  can  only  be  achieved  by  the  development  team. 

To  manage  quality  they  must 

•  have  control  of  their  process 

•  have  the  proper  data  to  track  quality 

•  be  properly  trained  and  motivated 

The  self-directed  team  management  style  empowers  the  team  to 
manage  quality. 

The  integrated  measurement  framework  provides  the  data. 

PSP  provides  the  training,  motivation,  and  commitment. 
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Topics 


Introduction 
TSP  Concepts 

•  Self-directed  teams  and  coaching 

•  Personal  Software  Process 

•  Process  and  measurement  framework 

•  Comprehensive  quality  management 

Team  management  with  TSP 

User  experience 
Getting  Started 
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Team  Management  with  TSP 


With  the  TSP  measurement  framework,  teams  know  exactly  where  they  stand 
in  several  dimensions. 

•  Schedule 

•  Resources 

•  Product  quality 

Teams  use  the  data  to 

•  manage  their  work 

•  anticipate  and  address  problems  early 

•  improve  cost,  schedule,  and  quality 

The  teams  and  their  managers  use  the  same  data  to  manage  the  project  as 
illustrated  in  the  following  sample  of  TSP  charts  and  forms. 
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Resource  Tracking 


Cumulative  plan  and  actual  resource  hours  shows  resource  burn  rate  and 
potential  source  of  slip 


1200.0 


1000.0 


800.0 


^  600.0 


J2 

D. 

<D 

> 


E 

3 

o 


400.0 


200.0 


0.0 


-  Cumulative 
Planned 
Hours 

-  Cumulative 
Actual  Hours 


Typical 

software 

project 


'^'^'^'^'^'^'^'^'^LOLOLOLOLOLOLOLOLO 

CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3 

CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ3CZ30CZ3CZ3CZ3CZ3 

CNCNCNCNCNCNCNCNCNCNCNCNCNCNCNCNCNCN 


CO  CN  CN 

^  a5  a5  o  o 


SN 


CNl 


CN 

CM 


Week 


CO 


_  CN  CN 

CM  CNl  CO  CO 


^  CN 
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Percent  Complete 


Earned  Value  Tracking 
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TSP  Weekly  Status  Report 


Plan/ 

Actual 

1.76 

0.96 

1.64 

0.94 

1.01 

0.96 


Assembly 

Phase 

Tasks  Completed  or  Due 

Resource 

Task  Plan 

Mrs. 

Task 

Actual  Mrs. 

Earned  or 

Plan  Value 

Planned 

Week 

Plan  vs. 

Actual  Mrs. 

Main  Form 

CODEINSP 

Main  Form  Code  Inspection 

SA 

1.5 

2.4 

0.1 

10 

0.63 

OEMMOO  Delivety.aspx 

UT 

OEMMOO  Delivery.aspx  (FE-Server)  U 

NK 

8.9 

3.0 

0.5 

13 

2.91 

OEMMOO  Deliverv.aspji 

DLDINSP 

OEMMOO  Delivery.aspx  (FE-Client)  Dl 

NK 

0.0 

0.0 

0.0 

13 

OEMMOO  Delivety.aspx 

CODE 

OEMMOO  Delivery.aspx  (FE-Client)  Ci 

NK 

7.5 

5.7 

0.4 

14 

1.32 

OEMMOO  Delivetv.aspji 

OR 

OEMMOO  Deiivery.aspx  (FE-Client)  Ci 

NK 

3.8 

1.7 

0.2 

14 

2.26 

OEMMOO  Delivei^.asp)! 

COMPILE 

OEMMOO  Deiivery.aspx  (FE-Client)  Ci 

NK 

1.3 

0.9 

0.1 

14 

1.44 

OEMMOO  Delivetv.aspji 

CODEINSP 

OEMMOO  Deiivery.aspx  (FE-Client)  Ci 

NK 

0.0 

0.0 

0.0 

14 

OEMMOO  Delivei^.asp)! 

UT 

OEMMOO  Deiivery.aspx  (FE-Client)  Ui 

NK 

5.9 

6.8 

0.3 

14 

0.87 

Query  Object 

TD 

Query  Object  Test  Development 

MB 

0.0 

0.0 

0.0 

14 

Query  Object 

CODEINSP 

Query  Object  Code  Inspection 

MB 

0.0 

1.2 

0.0 

14 

0.00 

1  1  ri  |-i  i  ri  r-+ 

1  IX 

r^|-iirir-+  1  li^i+Xrir-t  Pi  i --i  1  ri  rn  r- 

h;1D 

■1  -1 

■1  7 

n  -1 

■1  ;1 

n  cc 

TSP  Week  Summary  -  Form  WEEK 


Name 

Carol 

Date 

4/7/2003 

Team 

PSP  Ghost 

Status  for  Week 

15x 

Cycle 

Week  Date 

3/10/2003 

Weekly  Data 

Plan 

Actual 

Scheduie  hours  forthis  week 

151.0 

86.0 

Scheduie  hours  this  cycie  to  date 

1526.0 

1594.8 

Earned  vaiue  forthis  week 

6.9 

4.2 

Earned  vaiue  this  cycie  to  date 

79.5 

84.3 

To-date  hours  for  tasks  compieted 

1580.7 

1568.1 

To-date  average  hours  perweek 

101.7 

106.3 
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Cumulative  Defects  Removed  by  Phase 


Quality  Tracking 


Percent  Defect  Free 


o 

o 

-I 

4-* 

o 

0) 

0) 

Q 


Cumulative  Defects  Removed  by  Phase  for  Assembly  SYSTEM 


Defect  Density  by  Phase  for  Assembly  SYSTEM 


-♦ —  Ran 
-■ — Actual 
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Quality  Profile 


The  TSP  Quality  Profile  is  a  quality  early  warning  indicator. 

It  examines  criteria  that  are  effective  predictors  of  system  test  and  post-release 
quality,  and  produces  a  graph  of  the  result. 

It  supports  drill  down  to  any  level  for  further  analysis,  e.g.  in  software: 


system  ^  component  ^  module  ^  class. 


Quality  Profile  Criteria 


Quaiity  Profile 


1.  Design  time  =  coding  time 

2.  Design  review  time  =  design  time 

3.  Code  review  time  =  V2  coding  time 

4.  Compile  defects  <  10  per  KLOC 

5.  Unit  test  defects  <  5  per  KLOC 

If  satisfied,  a  criterion  has  a  value  of  1 ,  and  is  drawn 
along  the  outer  edge  of  the  chart. 


Design/Code 
Time  Ratio 


Code/Code 
Review 
Time  Ratio 


Unit  Test 
Defect  Density 


Compile  Defect 
Density 
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Using  the  Quality  Profile 
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Topics 


Introduction 
TSP  Concepts 

Team  management  with  TSP 
User  experience 
Getting  Started 
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The  Business  Case  for  TSP 


The  principal  cost  of  introducing  TSP  are  training  costs  and  lost 
opportunity  cost  resulting  from  time  spent  in  training. 

The  principal  benefits  are 

•  lower  development  costs  and  shorter  schedules 

•  more  functionality  per  release  and  improved  productivity 

•  lower  defect  density  in  both  system  test  and  in  the  delivered  product 

•  improved  work-life  balance  for  the  developers 

•  improved  customer  satisfaction 
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Schedule  Management 


First-time  TSP  projects  at  Microsoft  had  a  10  times  better  mean  schedule  error 
than  non-TSP  projects  at  Microsoft  as  reflected  in  the  following  table. 


Microsoft  Schedule  Results 

Non-TSP  Projects 

TSP  Projects 

Released  on  Time 

42% 

66% 

Average  Days  Late 

25 

6 

Mean  Schedule  Error 

10% 

1% 

Sample  Size 

80 

15 
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Managing  Task  Hours 


Task  hours  are  the  hours  that  teams  spend  on  planned  tasks  and  do  not  include 
unplanned  but  necessary  tasks  like  meetings,  courses,  coordination,  handling  mail,  etc. 

When  measured,  tracked,  and  managed,  the  team  can  usually  improve  task  hours,  but 
management  can’t.  Why? 
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Improving  Task  Hours 


At  Allied  Signal  average  task  hours  per 
developer  per  week  were  improved  from 
9.6  hours  to  15.1  hours  through  quiet  time, 
process  documentation,  more  efficient 
meetings,  etc. 


This  is  equivalent  to  a  57%  increase  in 
productivity. 


Software  Engineering  Institute 
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Actual  Task  Hours  per  Week 
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+57% 


-♦-Avg.Task  Hours -Week 
-i-Avg.Task  Hours -Phase 


Source:  Allied  Signal 
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Reviews  and  Inspections  Save  Time 


Xerox  found  that  TSP  quality  management  practices  reduced  the  cost  of  poor 
quality  by  finding  and  removing  defects  earlier  when  costs  are  lower. 
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Defect  Removal  Time  by  Phase 
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Removal  Phase 
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Intuit  Productivity  Improvement 


By  putting  a  quality  product  into  system  test  Intuit  improved  productivity  and 
reduced  cost  while  delivering  33%  more  functionality  than  planned. 


Results  at  Intuit:  Productivity 


■  During  2007  over  60%  of  Intuit’s  Small  Business 
Division  used  TSP 

■  TSP  was  a  major  contributor  to  the  QuickBooks  2007 
release 

■  It  was  the  smoothest  release  anyone  can  remember: 

■  On  time  delivery  of  all  planned  scope 

■  13  new  features  were  added  during  the  cycle(33% 
of  initial  scope) 

■  Saved  $700K  in  temporary  testing  staff  expenses 

■  Level  of  automated  testing  coverage  was  doubled 
compared  to  previous  year 


Focused  improvements  helped  deliver  a  great  release 

Source:  Intuit 
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Intuit  Quality  Improvement 


TSP  reduced  defects  found  in  system  test  by  60%  over  the  previous  two 
releases  of  QuickBooks  2007  release. 


Intuit  has  also  recently  reported  a  savings  of  $20M  from  a  reduction  in 
customer  support  calls  on  QuickBooks  2007. 

Results  at  Intuit:  Improved  Quality 


Cumulative  Dafectfi  Found  — acw  — 


III  2007  r^60°/o  fewer  defects  were  found 
in  System  Test  than  the  previaus  two  releases 


Source:  Intuit 
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Work-Life  Balance 


Finding  and  retaining  good  people  is  critical  to  long-term  success. 

Intuit  found  that  TSP  improved  work-life  balance,  a  key  factor  in  job 
satisfaction. 

Results  at  Intuit:  Improved  Work- Life  Balance 

■  Half  as  many  weekend  source  check-ins 
(<3%) 

■  Reduced  $  on  dinners  as  measured  by  P5S  - 
“Pizza  Slices  Served" 


12,000  pizza  slices 
served  last  year 


VS 


~30  pizza  slices 
this  year 


TSP  helped  improi^ed  employee  work  life  balance 


Source:  Intuit 


Topics 


Introduction 
TSP  Concepts 

Team  management  with  TSP 
User  experience 
Getting  Started 
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TSP  Product  Suite:  Process,  Training,  Tools 


Process  Notebook 

Process  scripts 
Forms 

Guidelines  and  standards 
Role  descriptions 
Training  and  Textbooks 
Executives 
Project  Managers 
Engineering 
TSP  Coach 
TSP  Trainer 

Tools 

.  TSP  Workbook 
.  PSP  Workbook 

Coach/T  rainer  Workbook 


::i _ 

TSP  Team  Launch  -  Script  LAU 


Purpose 

To  guide  teams  in  launching  a  soft^vare- intensive  project 

Entry  Criteria 

-  The  launch  prep  aration  woik  has  b  e  en  c  omplete  d  (T  REP  L,  P  REPT) . 

-  All  team  members  and  the  team  leader  are  committed  to  attend  launch 
meetings  1  through?  and  the  launch  postmortem,  and  management  and 
marketing  representatives  are  prepared  and  available  for  meetings  1  and?. 

-  An  authorize  d  launch  c  o  ach  is  on  hand  to  le  ad  the  launch  pro  cess. 

Day  I  1  I  2  I  3  I  4 


Introduction  to  the 


Introduction 
to  the 

Personal 

Software 

Process 


Team  Software 


A  Sclf-Improvcnicnt  Process 
for  Sof^are  Engineers 


Watts  S.  Hiunphrc 


■4t^ 
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TSP  Implementation  Strategy 


TSP  is  implemented  on  a  project-by-project  or  team-by-team  basis 

Start  with  two  or  three  teams. 

•  train  the  team  members  and  their  managers 

•  launch  these  teams  with  TSP 

•  evaluate  and  fine  tune  the  approach 

This  cycle  is  then  repeated,  increasing  scope  at  a  sustainable  pace. 
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Deployment  Timeline 


Task 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

TSP  Executive  Strategy  Seminar 

♦ 

Leading  Deveiopment  Teams 

♦ 

PSP  Fundamentais 

♦ 

Launch  Initial  Teams 

♦ 

Cycle  Postmortem  for  Initial  Teams 

♦ 

Re-launch  Initial  Teams 

♦ 

Train  instructors  and  coaches 

IIH 

Project  Postmortem  for  Initial  Teams 

♦ 

Train  and  launch  remaining  projects  and 

teams  at  a  sustainable  pace. 

■■ 

The  training  schedule  can  be  compressed  to  as  short  as  one  month  for  a  faster  start. 
The  gating  factor  for  most  organizations  is  the  availability  of  projects. 

SEI  recommends  training  internal  coaches  as  soon  as  possible. 
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Selecting  Pilot  Projects 


Pick  2  to  3  pilot  projects. 

•  3  to  15  team  members 

•  4  to  18  month  schedule 

•  software-intensive  new  development  or  enhancement 

•  representative  of  the  organization’s  work 

•  important  projects 

Select  teams  with  members  and  managers  who  are  willing  to  participate. 
Consider  the  group  relationships. 

•  contractors 

•  organizational  boundaries 

•  internal  conflicts 
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Build  Internal  Capability 


Organizations  should  develop  internal  capability  to  support  TSP. 

•  SEI-certified  TSP  coaches  are  essential 

•  SEI-authorized  trainers  are  optional  as  training  can  be  outsourced 

The  initial  pilot  projects  provide  the  “hands-on”  experience. 

•  first  SEI  leads  the  effort  and  internal  staff  observe 

•  then  internal  staff  lead  and  SEI  mentors 

Training  and  authorization  requirements 

•  Coach  -  one  week  training  course,  exam,  and  a  launch  observation 

•  Instructor  -  one  week  training  course  and  an  exam 
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Training  for  Participants 


Participant 

CBT 

Option 

Course 

Notes 

Executives  and 
senior  management 

No 

TSP  Executive  Strategy  Seminar 

1  day  +  optional  day  strategic  planning  session. 

Middle  and  first-line 
managers 

No 

Leading  Development  Teams 

3  days 

Software  developers 

Yes 

PSP  Fundamentals 

PSP  Advanced 

5  days 

5  days  (optional) 

Team  members 
other  than  software 
developers 

TSP  Team  Member  Training 

2.5  days  (will  replace  Introduction  to  Personal 

Process  in  2009) 

Instructors 

No 

PSP  Instructor  Training 

5  days 

Pre-requisite  training:  PSP  Fundamentals  and  PSP 
Advanced  or  PSP  1  and  PSP  II 

Coaches 

No 

TSP  Coach  Training 

5  days 

Pre-requisite  training:  PSP  Fundamentals  and  PSP 
Advanced  or  PSP  1  and  PSP  II 
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Questions? 
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The  IDEALSM 
Model 


Learning 


Analyze 

and 

Validate 


Propose 

Future 

Actions 


Implement 

Solution 


Refine 

Solution 


Pilot/Test 
I  Solution 


Desired 
;  States 


Create 

Solution 


Develop 

Recommen 

\dations 


Plan 

Actions 


Priorities  I  Develop  ’ 
I  Approach 


Stimulus  for^ 
Change 

/ 

\Set 

/Context 

Build 
Sponsor¬ 
ship  . 

/charter  / 

/  Infra-  / 

/  structure  / 

initiating  ! 

Characterize  I 
i  Currents  \ 

Diagnosing 


IDEAL  is  a  service  mark  of 
Carnegie  Mellon  University. 


Establishing 


Acting 
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Contact 

Awareness 

Understanding 

Trial  Use 

Adoption 

•Conversation 

•Conferences 

•Books 

•Org  Sponsorship 
(MSG) 

•Rollout  Strategy 

•Website 

•Article 

•Books 

•Articles 

•Training 

•Classes 

•Conferences 

•Consultants 

•Change  Agency 
(EPG) 

•Action  Teams 
(PATS) 

•Training 

•Support 

•New  Organizational 
Processes/Innovation 

•Pilot  Projects 
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The  Technology  Acceptance  Model  is  an  Information  systems  theory  that 
models  how  users  come  to  accept  and  use  a  technology. 
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Bagozzi,  R.  R,  Davis,  F.  D.,  &  Warshaw,  P.  R.  (1992).  Development  and  test  of  a  theory  of  technological 

learning  and  usage.  Human  Relations,  45(7),  660-686. 
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CMMI  for  Services:  An  Approach 
to  Improve  Your  Program 
Management  Office 


NDIA  CMMI  Technology  Conference 
November  17, 2009 
Pat  Mitryk 
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Topics 


•  Situation  Discussion 

-  Current  Situation 

-  Problems/Questions 

•  Scoping 

-  Challenges 

-  Before  We  Can  Get  Started 

-  Scoping  Decisions  and  Decisions 

•  Some  Steps  Toward  the  Improvement 

•  Benefits 

•  Lessons  Learned 
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Situation  Discussion 
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Current  Situation  -  What  Business  Are  We  In? 


Government  Contracts  for  a  variety  of  Strategic  Support 
Services 

-  IDIQ  -  Indefinite  Delivery,  Indefinite  Quantity 

-  Program  Support  &  Leadership 

-  Technical  Support  -  Engineering,  Communications,  IT, 
etc. 

-  Resource  sourcing  for  a  wide  variety  of  services 


ImpiuvtJ  uu5)iuiiit;i 
Satisfaction 


Improve  profitability  = 
Increase  Direct  Labor 


service  delivery  and  support  while 
reducing  costs 
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Typical  Make-up  of  PMO 


Current  Situation  -  2:  What  is  the  “PMO”? 


Program  Management  Office  Entities  -  Functional  Groups,  as 
typically*  defined... in  this/these  scenario’s 


Function 

Main  Responsibilities 

Program  Management 

Oversight  of  the  contract  vehicle  and  the  specific  Deliveries  or 
Taskings;  Main  customer  interface  for  satisfaction  and 
ongoing  business  development 

Project  Management 

Management  of  specific  Delivery  Orders  /  Task  Orders  and 
the  resources  that  perform  the  technical  and  management 
requests  of  the  customer 

Contracts 

Expert  in  government  contracts  establishment  and  change 
management  or  maintenance 

Pricing 

Pricing  for  all  direct  charges  to  the  original  contract  and  any 
change  orders 

Subcontracts 

Fulfillment  of  resource  requests  from  partner  companies  for 
expertise  in  support  of  Deliveries/Taskings 

Finance 

Establishment  and  change  management  of  budget;  Regular 
ongoing  performance  reporting  to  Program/Project 
Management  and  Customer(s) 

*Your  company  may  define  these  entities  differently,  or  combine  them;  they  are  here  to  assist  in  the 

depiction  of  this  CMMI-SVC  implementation  approach  /  example 
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Current  Situation  ■  2:  What  is  the  “PMO”? 


Program  Management  Office  Functional  Groups  -  continued 


Function 

Main  Responsibilities 

Acquisition/Procurement 

Materials  procurement  (and  management)  in  support  of 
Delivery/Task  Orders 

Facilities 

Ensuring  contract  resources  (onsite  and  govt  site)  have  office 
space  and  equipment;  Oversight  of  current  company  (onsite) 
facility  and  any  issues  or  requests 

Security 

Handles  security  clearance  and  security  logistics  and  access 
for  all  resources  both  here  and  abroad 

Recruiting 

Procurement  of  new  hires  to  fulfill  corporate  positions  and 
new  requests  for  contract  support 

Human  Resources 

Onboarding  and  training  for  resources 

Quality  Assurance 

Ensuring  quality  in  each  Functional  Group  and  Project 
processes/deliverables 

IT 

Computer  (Hardware  &  Software)  procurement,  configuration, 
and  support  for  onsite  resources 

1U18/2009  cognenceinc 
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Current  Situation  -  3 


Focus  on  “customer 
service”  without 
defining  “who  is  the 
customer” 


Accepting  Requests 
for  services  from 
any  source  in  any 
way 


PMO 

Operations 

Functional  Groups 


Lacking  communication 
&  coordination  with  one 
another 


Lacking 
understanding 
of  current 
performance 


Ignoring  linkage  Lacking  understanding  of 

to  business  goals  » “Project”  and  how  they 

relate  to  it 
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Current  Situation  -  4:  Typical  Problems 


People  Process  Technology 


•  Feel  like  everyone  comes  to 
them  with  a  business  or 
customer  “priority” 

•  Frustrated  at  their  jobs 

•  Not  fully  trained 

•  Work  as  firefighters,  not  as  a 
team 

•  Not  coached  or  mentored 

•  Lack  open  communications, 
respect 

•  Do  not  understand  where 
they  belong  -  under  an 
organization  and  structure 

•  Are  constantly  overburdened 


•  Are  not  using  a  well 
documented  and  repeatable 
standard  process 

•  Processes  that  exist  are  not 
deployed/trained  in  a  timely 
fashion  or  to  the  appropriate 
groups 

•  Little  or  no  SLA’s  and 
measures  are  in  place 

•  Continuous  improvement  is 
not  a  widely  understood 

•  QA  process  rarely  occurs  and 
perception  is  “policing”  and 
negative 

•  Best  practices  are  rarely 
developed,  and  not  typically 
shared,  and  refined 


•  Tools  exist,  but  are  not 
documented 

•  People  have  not  been  trained 
and  are  not  using  them 
consistently 

•  Requests,  Problems, 
Changes,  and  Issues  are  not 
captured,  tracked  and 
managed 

•  Not  easily  accessible  by 
people  when  out  of  the  office 
-  at  home  or  on  the  customer 
site 

•  Version  updates  are  pushed 
out  without  notification 
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Questions  that  Needed  Answers 


•  We  use  CMMI-SVC  Process  Areas  to  answer  the  following 
questions? 

-  What  are  the  requirements  for  the  PMO  and  all  functional  groups 
within  it? 

-  How  do  we  estimate  and  allocate  resources  for  a  project? 

-  How  do  we  measure  improvement? 

-  What  is  a  Service  Request  -  how  do  we  handle  each  one? 

-  What  can  we  promise  within  a  Service  Level  Agreement  to  the  end 
customer? 

-  What  is  our  current  capacity  and  availability  and  how  do  we 
estimate  and  forecast  -  for  “project”  work  and  new  proposals? 


11/18/2009  cognenceinc  SUdeO 

©2002-  cognence,  inc.  Improving  Software  Economics 


Scoping 


r 


L  J 


cognenceinc 

Improving  Software  Economics 

©2006  -  cognence,  inc. 


Scoping  Challenges 


Typical  scenario  -  everyone  is  maxed  out  so.... 

•  Where  do  we  begin? 

-  It’s  a  big  model,  what  are  our  most  important  issues  or  business 
objectives? 

-  It’s  a  big  company,  where  are  we  going  to  focus  our  initial  efforts? 

-  How  much  improvement  is  too  much? 

-  How  can  we  achieve  (and  show)  smaller  and/or  shorter  term 
success? 

•  How  should  we  organize  the  improvement  infrastructure? 

•  Who  should  be  involved  in  the  improvement  strategy,  plan,  creation 
and  implementation? 

•  How  do  we  get  started? 
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Getting  Started  - 1:  Define  Basic  Terminology 


We  could  not  begin  until  we  made  sure  we  had  agreement  on  some  basic 
common  terminology: 

Program*  -  a  collection  of  related  projects  and  the  infrastructure  that  supports 
them,  including  objectives,  methods,  activities,  plans  and  success  measures. 

In  our  language  -  “The  Award”  -  The  Contract  Vehicles  that  were  proposed 
outlining  management  objectives  and  techniques  for  the  customer 

♦  Contract  level  (very  high  level)  requirements  and  management  activities 

♦  Success  criteria/measures  -  but  no  actual  funding,  but  a  bucket  of  $$ 

Project*  -  a  managed  set  of  interrelated  resources  that  delivers  one  or  more 
products  or  services  to  a  customer  or  end  user.  In  our  language  -  the 
awarded  Delivery/Task  Orders  within  a  Program/Contract  Vehicle  that 
outlines  specific  work,  resources  and  deliverables  and  is  “funded” 

♦  Task  Order  level  requirements  for  resources  and  services 

♦  PMO  related  schedule  of  status  and  performance 

*CMMI-SVC  Glossary 
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Getting  Started  -  2:  Define  Basic  Terminology 


Requirements*  -  a  condition  or  capability  needed  by  and  end  user 
to  solve  a  problem  or  achieve  an  objective;  a  condition  or 
capability  that  must  be  met  or  possessed  by  a  product,  service 
to  satisfy  a  supplier  agreement,  standard,  specification,  or  other 
formaily  imposed  documents. 

Represented  by  contractuai  agreements  such  as: 

•  Technical  Execution  Plan  (TEP  or  RTEP) 

•  Program/Project  Work  Statement  (PWS) 

•  Statement  Of  Work  (SOW) 


*  CMMI-SVC  Glossary 

11/18/2009  cognenceinc  SUdelS 

©2002-  cognence,  inc.  Improving  Software  Economics 


Getting  Started  -  3:  Define  Basic  Terminology 


Sorvic©  R6C|U6St*  -  a  communication  from  a  customer  or  end  user  that  one 
or  more  specific  instances  of  service  delivery  are  desired,  in  our  ianquaqe, 
they  are: 

•  Task  Order  Requirements 

•  Requests  for  resources  (a  “body”  to  fill  a  Labor  Category) 

•  Requests  for  management  services  for  the  contract  (e.g., 
regularly  scheduled  meetings  and  management  reports) 

•  Contract  Modification  Requests 

•  internally  generated  requests  /  changes 

•  Request  for  proposal 

Note:  A  request  typically  requires  many,  if  not  all  functional  groups  (Pricing, 

contracts,  finance,  subcontracts,  recruiting,  etc.)  to  play  a  role  in  providing  the 
Program/Project  managers  with  the  information  and  deliverables  they  need  to 
answer  the  customer  request 

*CMMI-SVC  Glossary 
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Scoping  Decisions  -  Where  Do  We  Begin? 


Divide  and  Conquer!!! 

1 .  Improvement  in  Phases 

-  Phase  1 :  “Above  the  line”  -  PMO  Operations  (on-site  at 
company  x) 

•  Will  allow  us  to  control  two  factors: 

Customer  Satisfaction  -  Many  contracts  won  initially  and  on  re¬ 
compete  with  how  we  can  manage  the  contract  activities  to  maintain 
and  improve  customer  satisfaction 

Deliverable  control  -  we  can  show  shorter-term  success  by 
separating  “controliable”  groups  and  activities  from  “uncontrollable” 
ones  -  IDIQ-based  projects 

-  Phase  2:  “Below  the  line”  -  Project  team  processes  (customer 
site  and  typically  IDIQ  related  activities),  some  covered  by 
CMMI-DEV 
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r 

Scoping  Decisions  :  The  Organizational  Unit 
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Scoping  Decisions  -  Who,  What,  How? 


Divide  and  Conquer!!! 

2.  To  utilizes  our  prior  experience  with  CMMI-DEV, 
structure  (or  keep)  a  similar  PI  infrastructure 

-  Divide  the  improvement  activities  into  PMO  Entities  - 

“Functional  Groups”  (e.g,  Program/Project  Management,  Finance,  Pricing, 
Subcontracts,  etc.); 

•  Utilize  a  familiar  improvement  infrastructure 

•  Utilize  the  FG  leads  as  EPG  members 

•  Design  PATs  around  those  functional  groups 

3.  Enables  Functional  Groups  (FGs)  to  be  autonomous  in 
many  of  their  own  improvement  activities  while 
contributing  to  Organizational  Unit  level  improvement 
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Scoping  Decisions  -  Who,  What,  How? 


CMMI-SVC:  Performance 
Improvement  Team  Structure 


Executive 

Steering 

Committee 


Management  Team 


Services 

Performance  Improvement 
Team 


Formal  PI  Infrastructure: 

•  Charters 

•  Some  Common  Measures 

•  Regularly  Scheduled  Meeting 

•  Hard  Deliverables/Action  Item 
Tracking 

•  Weekly/Monthly  Status  Reports  to 
ESC/Sponsor 


Facilities  Subcontracts  Contracts  Pricing  Finance  Security  Acquisition  /  program  Mgmt 

Lead  Lead  Lead  Lead  Lead  Lead  Purchasing 

Lead 
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Position  for  Performance  Improvement 


Ongoing  Performance  Improvement  -  Simple  Flow 


Changes 

And 

Improvements 


Organization’s 
Standard  Process 
And  Assets 


Senior  Management, 
Services  Performance 
Improvement  Team 
(SPIT) 


Performance 

Analysis 


Performance 

Results 


Historical 
Data 

as  input  to 
Estimation  &  Tailoring 
(for  each  new  contract) 


Functional  Group’s 
Defined 
Process 


J  Actuals,  Results 
Lessons  Learned 
Reviews  &  Audits 
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Slide  19 


Some  Steps  Toward  Improvement 


cognenceinc 

Improving  Software  Economics 

©2006  -  cognence,  inc. 


step  1  -  Identify  the  Target 


Collect  baseline  data 

•  Perform  a  baseline  appraisal  -  SCAMPI  C  (or  SCAMPI  B)  using  SPIT 
and/or  PAT  team  members  -  this  front-end  investment  will  pay  off  over  the 
long  run 

•  Increased  organization  knowledge  of  what  is  required  and  why 

•  Assistance  in  breaking  down  barriers  to  improvement  activities 

•  If  the  appraisal  does  not  include  interviews,  hold  meetings  or  focus 
groups  or  do  a  survey  to  understand  all  the  current 
issues/frustrations/”pain  points”  -  from  everyone’s  perspective 

•  Collect  any  measures  of  current  performance  -may  require  some  data 
mining  or  a  Cost  of  Quality  Appraisal/Project  Retrospective 
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step  2  -  Strategize  the  Transformation 


strategy/ 

Planning 


CMMI 

Baseline 

Training 

Appraisal 

AT 

CoQ 

Training 

Appraisal 

Tools 

Appraisal 


Personnel 

Appraisal 

Project 

Review 


SPIT 

Workshop 

Planning 

Sessions 

1 

CMMI  Process 
Implementation 

■ 

Internal  QA  Audits 

■ 

Tools 

Implementation 

Mock 
SCAMPI  A 

■ 

SCAMPI 

A 

Training 


APEX  Transformation  Methodology  -  by  cognence 
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step  2  -  Strategize  the  Transformation 


Why? 

•  To  rapidly  collect  appropriate  levels  of  information  and 
identify: 

•  Opportunities  for  improvement 

•  Barriers  and  risks  to  the  improvement  effort 

•  To  understand  the  As-ls  -  current  health  and  maturity  levels 

•  To  determine  the  To-Be  -  improvement  that  can  be  achieved 
through  application  of  best  practices 

•  To  identify  short  term  wins  and  long-term  direction 
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step  3  -  Approach  from  Top-Down  &  Bottom-Up 


Multi-dimensional  nature  of  implementation  includes  definition  at 
EPG/SPIT  level  as  well  as  each  Functional  Group  level 

1.  SPIT  defines: 

•  High  level  strategic  items  (driven,  with  input  from  the  Executive  Steering 
Committee) 

•  Needs,  capabilities,  objectives 

•  External  Customer(s) 

•  Common  measures  -  customer  facing  SLAs 

•  Common  tools 

•  Overall  planning  and  implementation  approach 

•  Integrated  training  materials  and  plans 


2.  Each  Functional  Group  defines: 

•  Specific  items  that  define  Functional  Group  operational  processes 

•  Specific  training  -  for  Functional  Groups  members  and  for  interfacing  FGs 
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step  3  -  Example:  Service  Delivery 


1.  SPIT  defines: 

•  Service  Request  from  the  customer  or  customers  to  projects 

•  General  guidelines  for  identifying  and  managing  service  requests 

•  High  Level  Flow  for  each  request  type 

2.  Each  Functional  Group  defines: 

•  FG  flow  for  each  request  type 

•  Process  and  tools  for  identifying  and  managing  each  request  type 

•  Identification  of  any  additional  requests  (that  may  be  particular  to  the 
FG) 
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step  3  -  Example:  Measurement  &  Analysis 


1 .  SPIT  defines: 

♦  Business  Objectives 

♦  Common  measures  to  support  business  objectives 

♦  Measurements  reports  provided  from  the  PMO  to  the  Customer(s) 

•  Aggregation  of  Functional  Group  Measures  and  other  Contract  supporting 
measures 

♦  Integrates  Functional  Group  measures  into  Customer  reports 

2.  Each  Functional  Group  defines: 

♦  Specific  measures  that  will  trace  FG  process  to  overall  business  objectives 

♦  Measurements  provided  to  the  organization  (internally)  e.g., 

•  Request  response  time 

•  Request  throughput 

♦  Details  of  collection,  storage  and  analysis  processes 
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step  3  -  Example:  Configuration  Management 


1.  SPIT  defines: 

•  Organization  Repository 

•  Generai  change  controi  guideiines 

2.  Each  Functional  Group  defines: 

•  Their  group  Ci’s  -  documents,  deliverable  and  internal 

•  Specific  storage,  archival  and  access 
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step  3  -  Example:  Strategic  Service  Management 


1 .  SPIT  defines: 

•  Business  Objectives 

•  Business  Capabilities,  Strategic  Needs 

♦  Customers  (External) 

♦  Services  provided  from  the  PMC  to  the  Customer(s)  -  catalog  of  services 

2.  Each  Functional  Group  defines: 

♦  Customers  -  which  include  other  Functional  Groups  (Internal  &  External) 

•  Functional  Group  Capabilities 

♦  Functional  Group  Business  Process  Flow 

•  Specific  FG  services  -  cataloged 
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step  3  -  Example:  Capacity  &  Availability  Management 


1. 


SPIT  defines: 

•  General  guidelines  for  establishing  a  strategy  for  capacity  and 
availability 

•  Guidelines  for  aggregating  measures  to  manage  (analyze  and  report) 
capacity  and  availability 


2.  Each  Functional  Group  defines: 

•  Specific  strategy  to  handle  FG  capacity  and  availability 

•  Estimation  methods  (used  for  PP  as  well)  for  pricing 

•  Specific  measures  and  tools  for  monitoring  and  reporting  FG  capacity 
and  availability 
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step  4  -  Show  The  Benefits 


•  Sample  Performance  Improvement  Success  Story 

1 .  Development  of  measures  and  standard  processes  for  the  way  they  deliver 
services  support  within  their  Functional  Group  -  handling  requests 

2.  They  measured  their  service  request  cycle  times  and  have  made  significant 
improvement: 

Reductions  in  cycle  times: 

31%  -  Subcontract  Processing 
^  69%  -  Modification  processing 
^  85%  -  Task  Order  Processing 
Increase  in  Capacity 
350% 


Throughpu 

TAT  (#  Days) 

2008 

2009 

2008 

2009 

Subcontracts 

69 

182 

58 

39 

Modifications 

979 

3513 

16 

5 

Task  Orders 

171 

624 

17 

3 

Totai  Capacity 

1219 

4319 
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Slide  30 


step  4  -  Show  The  Benefits 


other  Benefits: 

•  The  use  of  CMMI-SVC  enabled  us  to  do  what  the  CMMI-DEV 
did  not  -  use  (i.e.,  manage)  the  “Service  Request” 

•  Top-Down,  Bottom-Up  Improvement  allows  folks  at  various 
levels  of  the  organization  some  “say-so”  which  achieves  buy-in 

•  Once  the  folks  who  participate  in  providing  the  service  know 
how  they  relate  to  the  end  customer  as  well  as  their  direct 
customers,  request  handling  is  much  more 
simplified/understood  and  easier  to  measure 

•  Provides  functions  autonomy  as  well  as  participation  in  the 
overall  business  performance  improvement  -  effective  service 
delivery  to  the  end  customer 
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step  5  -  Learn  Your  Lessons 


“First  Time  “Victim”,  Second  Time  “Volunteer”! 

•  Not  defining  the  basic  terminology  first 

And  don’t  forget  to  identify  the  “customer”  and  the  “requests” . all  of  them 

•  Not  keeping  to  a  “KISS”  principle 

Focus  on  simple  processes  and  put  more  effort  into  training  and  mentoring  through 
deployment 

•  Taking  on  too  much  too  soon 

Put  a  realistic  schedule  into  place  and  account  for  already  overtaxed  resources 

Look  for  small,  but  meaningful  wins  (use  the  FG  that  the  organization  complains  about 
the  most  to  show  improvement)  to  get  the  most  “converts” 

•  Not  cataloging  resistance  factors,  risks,  issues 

Keep  a  centralized,  easily  accessible  repository  so  Individuals  and  Functional  Group 
PATs  can  document  items  and  EPG/SPIT  can  work  to  resolve  them 

•  Too  much  multitasking  -  attempting  to  do  process  in  the  margins 

Outline  a  spend  plan,  track  it,  report  it  and  get  management  to  act 

•  Insufficient  membership  and  skills 

Ensure  process  improvement  teams  have  adequate  skill  sets 
“Seed”  the  organization  with  folks  who  will  carry  the  “banner” 

•  Failure  to  maintain  momentum  (i.e.,  visibility) 

Weekly  and  Monthly  Status 

11/18/2009  cognenceinc  SUdeSZ 

©2002-  cognence,  inc.  Improving  Software  Economics 


Questions 


Contact 


Pat  Mitryk 

cognence,  inc. 

Improving  Software  Economics 
www.coqnence.com 
pat  mitrvk@coqnence.com 

732.804.6410 
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Improvement 

and 


Understanding  the 
Application  of 
SCAMPlsi^  Method  to  it 


opening 

Recap  High  Maturity  Process  Areas 

Main  Questions  for  High  Maturity  Process 
Improvement 

Pilot  Lessoned  Learned 


Agenda  and  Topics 


CMMI  ML  4  &  5  PAs  Recap 


6  Organizational  Process  Performance 
Quantitative  Project  Management 
Causal  Analysis  and  Resolution 
Organizational  Innovation  and  Deployment 


specific  Practices  of  OPP 


SG  1  Establish  Performance  Baselines  and  Models 
SP  1 . 1  Select  Processes 

SP  1 .2  Establish  Process-Performance  Measures 

SP  1.3  Establish  Quality  and  Process-Performance 
Objectives 

SP  1 .4  Establish  Process-Performance  Baselines 
SP  1.5  Establish  Process-Performance  Models 


Establish  Performance  Baselines  and  Models 


Establish 

Process- 

Performance 

Measures 


Select 

Processes 


Selected  Subprocesses  from 
Org.  Std.  Processes 


Establish 

Process- 

Performance 

Baselines 


Organization’s  Set  of 
Standard  Processes 


Establish 

Process- 

Performance 

Models 


Organizational  Process' 
Performance  Baselines 


Establish  > 
Quality  and 
Process- 
Performance 
Objectives 


Organization’s  Quality 
and  Process-Performance 
Objectives 


Process- 

Performance 

Models 


Organizational  Process 
Performance  Context 


MA 


QPM 


pThe  first  three  SPs  establish  proeesses  (subprocesses), 
measures,  and  objectives  at  the  organization  level  that  focus 
and  align  the  quantitative  management  activities  of  projects 
(QPM)  with  the  business  objectives  of  the  organization. 

last  two  SPs  take  the  actual  results  obtained  from 
projects  to  create  baselines  and  models  that  enable  the  next 
project  to  predict  what  performance  to  expect  from  selecting 
certain  subprocesses  for  its  use,  and  thereby  assess  its  ability 
to  meet  its  objectives. 


Specific  Practices  of  QPM 


SG  1  Quantitatively  Manage  the  Project 

SP  1 . 1  Establish  the  Proj  ect’  s  Obj  ectives 

SP  1 .2  Compose  the  Defined  Process 

SP  1.3  Select  the  Subprocesses  That  Will  Be  Statistically  Managed 
SP  1.4  Manage  Project  Performance 
SG  2  Statistically  Manage  Subprocess  Performance 

SP  2. 1  Select  Measures  and  Analytic  Techniques 
SP  2.2  Apply  Statistical  Methods  to  Understand  Variation 
SP  2.3  Monitor  Performance  of  the  Selected  Subprocesses 
SP  2.4  Record  Statistical  Management  Data 


Project’s  Quality  and 
Process-Performance 
Objectives 


Establish 

Project’s 

Objectives 


Compose 
the  Defined 
Process 


Remedial 

Actions 


Select  the 
Subprocesses 
that  Will  Be 
Statistically 
Managed  . 


Manage 

Project 

Performance 


Statistically  Manage  Subprocess  Performance 


f  Apply 
Statistical 
Methods  to 
Understand 
Variation 


f  Monitor  ' 
Performance 
of  the  Selected 
Sub¬ 
processes 


Record 

Statistical 

Management 

Data 


Select 
Measures 
and  Analytic 
Techniques 


uantitati  ve  Proj  ect  Management 

Context 


Quantitatively  Manage  the  Project 


IPM 


Selected 

Subprocesses 


Predictions  of 
Quality  and 
Process 
Performance 


:ation’s 

•ement 
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QPM  Summary 


jcQPM  involves  both  quantitative  and  statistical  management.  The  project 

•  establishes  quantitative  objectives  based  on  the  organization’s 
business  objectives  and  needs  of  the  customer 

•  composes  a  defined  process  based  on  historical  capability  data  that 
will  help  it  meet  those  objectives 

•  monitors  the  project  quantitatively  to  assess  whether  the  project  is  on 
course  to  achieve  its  objectives. 

oFor  each  subprocess  to  be  statistically  managed, 

•  objectives  are  established  for  its  process  performance 

•  its  variation  is  understood  (subprocess  is  stable) 

•  when  the  subprocess  fails  to  achieve  its  objectives,  corrective  action  is 
taken 


SG  1 


SG2 


Specific  Practices  of  CAR 

Determine  Causes  of  Defects 
SP  1 . 1  Select  Defect  Data  for  Analysis 
SP  1 .2  Analyze  Causes 
Address  Causes  of  Defects 
SP  2.1  Implement  the  Action  Proposals 
SP  2.2  Evaluate  the  Effect  of  Changes 
SP  3.2  Record  Data 


ausal  Analysis  and  Resolution 

Context 


CAR  Records 


Address  Causes 
of  Defects 


Selected  Action 
Proposals  & 
Improvement 
Proposals 


Performance 

Measures 


GAR  has  its  greatest  value  when  performed  in  the 
context  of  a  quantitatively  managed  process. 

■  CAR  involves 

•  a  selection  of  defects  or  problems  whose  resolution 
would  benefit  the  organization 

•  a  root  cause  analysis 

•  development  and  implementation  of  an  action  plan  to 
remove  the  root  causes  of  the  defects  or  problems 


specific  Practices  of  OID 


SG2 


SG  1 


Select  Improvements 

SP  1 . 1  Collect  and  Analyze  Improvement  Proposals 
SP  1 .2  Identify  and  Analyze  Innovations 
SP  1.3  Pilot  Improvements 
SP  1 .4  Select  Improvements  for  Deployment 
Deploy  Improvements 
SP  2.1  Plan  the  Deployment 
SP  2.2  Manage  the  Deployment 
SP  2.3  Measure  Improvement  Effects 


Collect 
and  Analyze 
Improvement 
Proposals 


Identify 
and  Analyze 
Innovations 


Pilot  Evaluation  Reports 
Lessons  Learned 


Candidate  Innovative 
Improvements 


Process  and  Technology 
Improvement  Proposals 


Updated  Training 
Materials 


eploy  Improvements 


Deployment  Plan 


Measurement  Result: 


Measure 

Improvement 

Effects 


Plan  the 
Deployment 


Manage  the 
Deployment 


Updated  Processes 


Organizational  Innovation  and 
Deployment  Context 


Select 

Improvements 

for 

Deployment . 


Pilot 

Improvements 


cOID  uses  the  quantitative  information  developed  at  ML4  to 
identify,  analyze,  and  select  incremental  and  innovative 
rovements  to  the  organization’s  processes  and 
echnologies. 

cOID  involves  both  incremental  improvement  (everyone  in 
the  organization  is  involved)  and  revolutionary  improvements 
(outward  looking  and  opportunistic)  to  targeted  processes. 

Qimprovements  are  introduced  systematically  in  the 
organization  by  conducting  pilots,  analyzing  costs  and 
benefits,  and  planning  and  managing  deployment. 

cOID  embodies  continuous  improvement  that  results  from 
implementing  all  the  PAs  in  the  model. 


Main  Questions  for 
High  Maturity  Process 
improvement 


Main  Questions  for  High 
Baturity  Process  Improvement 

Are  able  to  determine  which  processes  / 
subprocess  are  suitable  to  be  measured 

•  **  consideration  note  -  selection  of  one  process, 
measure,  or  objective  will  constrain  the  selection  of  the 
others  ** 

Do  we  know  which  measures  are  useful  for 
determining  process  performance 

Do  we  have  quality  and  process-performance 
objectives  for  those  processes 


Main  Questions  for  High 
Baturity  Process  Improvement 

B  lIDo  we  have  the  skills  to  statistieally  measure, 
analysis,  communicate  and  act  according  to  the 
numbers,  what  additional  training  we  will  need 

Do  we  have  relevant  historical  data  (at  least  7~13 
points)  that  enable  us  to  establish  baseline  and 
trends 


Main  Questions  for  High 
Baturity  Process  Improvement 

Do  we  have  the  culture  on  identifying  causes  of 
defects  and  other  problems  and  take  action  to 
prevent  them  from  occurring  in  the  future.  And  in 
what  level 

Do  we  have  the  culture  or  the  capability  to  plan 
develop  and  deploy  incremental  and  innovative 
improvements  that  measurably  improve  the 
organization’s  processes  and  technologies 


upporting  the  SCAMPI  Process 

for  High  Maturity 

The  current  experience  shows  that  organizations 
that  strive  towards  process  improvements  that 
targeting  high  maturity,  do  not  always  understand 
the  model  expectations  and  requirements. 

When  we  add  to  it  the  abstract  level  of  the  high 
maturity  PAs,  we  are  creating  real  challenge  to  the 
quality  engineers  and  managers  capability  to 
support  the  business  improvements. 


upporting  the  SCAMPI  Process 

for  High  Maturity 


Based  on  these  observations  and  lessons  learned  and  as 
part  of  these  pilots  we  have  developed  a  number  of 
incremental  papers  and  guide  books  with  the  intent  to  build 
an  accumulative  knowledge  and  capabilities  to  interpret 
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Purpose 


The  purpose  of  this  presentation  is  to  propose  a  way 
to  address  GP  3.2  that  will  benefit  both  projects  and 
the  organization. 
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What  is  GP  3.2? 


GP  3.2  Collect  Improvement  Information 

□  Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and 
performing  the  process  to  support  the  future  use  and 
improvement  of  the  organization’s  processes  and 
process  assets. 


Remember  to  read  the  whole  practice! 
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Why  collect  Improvement  information?-1 


GP  3.2  Collect  Improvement  Information 

□  Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and 

performing  the  process  tO  SUppOlt  the  future 

use  and  improvement  of  the 
organization’s  processes  and  process 
assets. 


Satisfying  6P  3.2  shouldn't  be  your  purpose 
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Why  collect  Improvement  information?-2 


GP  3.2  Collect  Improvement  Information 


a 


Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and 

performing  the  process  tO  SUppOlt  the  future 

use  and  improvement  of  the 
organization’s  processes  and  process 
assets. 
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Reason  #1:  to 
support  future  USE 
of  the  processes 
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Why  collect  improvement  information?-3 


GP  3.2  Collect  Improvement  Information 

□  Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and 

performing  the  process  tO  SUppOlt  the  future 

use  and  improvement  of  the 
organization’s  procef  \es  and  process 
assets.  /  \ 


Reason  #2:  to  support 
future  IMPROVEMENT 
of  the  processes 
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Where  does  the  improvement  information  come  from? 


GP  3.2  Collect  Improvement  Information 

□  Collect  work  products,  measures,  measurement  results, 
and  Improvement  information  ClGrlVGCl  frOITl 

planning  and  parforming  tha  procass 

to  support  the  future  use  and  improvement  of  the 
organization’s  processes  and  process  assets.  / 
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Improvement  information  is  based  on 
the  actual  USE  of  the  process,  not  just 
what  the  Process  Group  dreamed  up! 
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Isn’t  Lessons  Learned  enough? 


Certainly  Lessons  Learned  can  support  future  use 
and  improvement  of  processes,  but  only  if  they  are 
properly  indexed  and  translated  into  process 
improvement  proposals. 

Otherwise,  teams  get  tired  of  poring  over  endless 
lists  of  irrelevant  lessons  learned  and  abandon  them 
altogether. 


It's  unreasonable  to  think  every  project  will  have  a 
lessons  learned  for  every  process  area. 
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What  else  do  we  need? 


GP3.2  Collect  Improvement  Information 

a  Collect  work  products,  measures, 
measurement  results,  and  improvement 

information  derived  from  planning  and  performing  the 
process  to  support  the  future  use  and  improvement  of  the 
organization’s  processes  and  process  assets. 


The  practice  says  we  should  collect  these  things  too! 
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Why  collect  work  products? 


□  Use  them  as  “examples”  to  speed  up  creation  of  similar 
work  products  on  future  projects 

□  Illustrate  special  cases  -  again  to  reuse  them  on  future 
projects 

□  Create  templates  from  new  or  unique  work  products  to 
make  the  work  easier  for  everyone 

□  Analyze  for  potential  process  trends  or  changes 


Support  future  USE  and  IMPROVEMENT  of  processes 
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What  kind  of  work  products  might  we  coilect?-1 


Project  Management  work  products: 


□  Project  Plans  (for  reuse  on  similar  projects,  to  improve 
planning  templates) 

□  Risk  registers  (to  identify  new  risk  sources,  categories, 
response  approaches,  recurring  risks) 

□  Estimates  (to  identify  new  or  better  work  product  or  task 
attributes) 


□  Miiestone  and  Progress  Briefs  (for  reuse  on  similar 
projects,  to  identify  recurring  issues) 
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What  kind  of  work  products  might  we  coilect?-2 


Engineering  work  products: 

□  Requirements  specifications  /  operationai  concepts 

(for  reuse  on  similar  projects  or  products) 

a  Design  specifications 

□  Code,  drawings,  data 
a  Integration  plans 

□  Installation  procedures 

□  Test  plans,  procedures  and  data 
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What  kind  of  work  products  might  we  collect?-3 


other  work  products: 

□  Contract  examples  and  evaluation  criteria  (for  reuse 
on  future  contracting  efforts;  to  improve  the  quality  and 
effectiveness  of  contracts) 

□  PPQA  non-compiiance  data  (to  identify  recurring 
process  issues) 

□  Decision  records  (to  improve  the  decision  making 
process  or  decision  criteria,  for  making  similar  decisions 
in  the  future) 

□  Project  taiioring  records  (to  identify  new  tailoring 
criteria,  new  life  cycles,  tailoring  trends,  work 
environment  adaptations) 
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What  happens  to  the  work  products  we  collect? 


Work 

Products 


Categorized 
Examples 
added  to  PAL 


Process 

Improvement 

Proposals 


Support  future  USE  and  IMPROVEMENT  of  processes 
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Measures  versus  Measurement  Results 


Why  distinguish  between  the  two? 


□  Measures  are  data  (a  data  point  or  chart) 

□  Measurement  results  are  information  (includes  analysis 
and  actions  taken) 
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Why  collect  measures  and  measurement  results? 

What  are  we  going  to  do  with  them? 

□  Use  for  estimating  for  similar  project  types 

□  Improve  measurement  analysis  and  reporting  procedures 

□  Improve  decision  process  /  criteria  and  provide 
quantifiable  data  on  which  decisions  can  be  based 

□  Predict  process  performance  in  future  (establish  process 
performance  baselines  and  models) 


Support  future  USE  and  IMPROVEMENT  of  processes 
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What  kind  of  measures  might  we  coiiect? 


□  Estimates  (for  reuse  on  comparable  efforts) 

□  Actual  performance  (to  improve  estimating  and  establish 
performance  baseiines) 

□  Unique  project-specific  measures  with  potentiai  for  reuse 

□  Planning  data  (from  GP  2.2) 

□  Monitoring  data  and  results  (from  GP  2.8) 


Support  future  USE  and  IMPROVEMENT  of  processes 
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What  happens  to  the  measures  we  collect? 


Measures  and 
Measurement 
Results 


Repository  with 
Categorized 
Historic  Data 


Process 

Improvement 

Proposals 


Process 
Performance 
Baselines  and 
Models 


Support  future  USE  and  IMPROVEMENT  of  processes 
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Why  is  there  a  person  in  the  picture? 


Subject  matter  experts  need  to  periodically  review, 
analyze  and  categorize  the  work  products  and 
measures  submitted  in  their  areas  of  expertise 
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Why  not  collect  everything? 

Although  the  CMMI  does  NOT  require  you  to  identify 
the  specific  work  products,  measures,  measurement 
results  and  improvement  information  that  you  will 
collect,  doing  so  can  have  benefits: 

□  Teams  know  what  to  collect 

□  Teams  know  where  to  find  and 
how  to  use  what  others  have 
collected 
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Back  to  Lessons  Learned-1 


Lessons  Learned  are  an  essential  aspect  of  GP3.2,  but  the 
results  need  to  be  indexed  so  teams  can  find  relevant  lessons 
learned  quickly.  Consider  indexing  by: 

□  Product  or  Product  Type 

□  Customer  or  Customer  Type 

□  Project  or  Project  Type  (Life  Cycle,  Fixed  Price  or  T&M) 

□  Process  during  which  the  lessons  learned  was  encountered 

□  Life  Cycle  phase  in  which  the  lessons  learned  was  encountered 

□  Life  Cycle  phase  in  which  the  lessons  learned  should  be  implemented 

□  Date  Lessons  Learned  was  encountered 

□  Whether  others  have  found  the  Lesson  Learned  useful 
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Back  to  Lessons  Learned-2 


Focus  on  incorporating  lessons  iearned  back  into  your 
processes.  Once  a  iessons  iearned  has  been  incorporated  into 
a  process,  it  can  be  archived  from  the  iessons  iearned 
database. 
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Three  Views  of  GP  3.2 


Let’s  look  at  GP  3.2  from  3  perspectives: 

□  Project  Planning  View 

□  Project  Execution  View 

□  Process  Management  View 
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Process  Asset  Library 


Planning  View 


Life  Cycles,  Tailoring 
Criteria,  Work 
Environment  Standards 


Organizational  Set 
Of  Standard  Processes 


Planning  Process 
Assets 


Sample  Assets  from 
Similar  Projects 


Project  Planning 

(PP. 

IPM  5P  1.2, 
IPM  5P  1.6, 
GP  3.2) 


Lessons  Learned 


Contributed  Planning 
Related  Work 
Products 


Measurement  Repository 


Historic  Oata  with 
Work  Product  /  Task 
Attributes 
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Project  Execution  View 


Process  Asset  Library 


Process  Assets 


Sample  Assets  from 
Similar  Projects 


Project  Execution 
(PMC,  &P  3.2, 
IPM  SP  1.5, 
IPM  SP  1.6) 


Lessons  Learned 


Contributed  Work 
Products 


Measurement  Repository 


Actual  Performance 
Data 
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Process  Management  View 


Process  Improvement 
Proposals 


Project  Work  Products 


Project  Measures  and 
Measurement  Results 


Process  Focus  Work 
Products 
and  Measures 


Process 
Management 
(OPF,  OPb.  OT. 
GP  3.2) 


\ 


Training  Plan  Updates 


Lessons  Learned 


(OPU  SGI)  Updates  to: 
•Standard  Processes 
•Lifecycle  Models 
•Tailoring  Criteria 
•Measurement  Repository 
•Process  Asset  Library 


PPQA  Plan  Updates 
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What  about  Organizational  GP  3.2s? 


Remember  to  implement  GP  3.2  for  process  management: 

□  Collect  work  products  such  as  training  material  examples, 
process  action  plans,  appraisal  plans  and  results,  process 
deployment  plans,  process  pilot  feedback 

□  Collect  measures  such  as  process  adoption  measures,  planned 
and  actual  effort  spent  on  process  improvement  activities, 
training  survey  results 

□  Collect  lessons  learned  at  defined  times  in  the  process 
improvement  life  cycle 
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How  can  I  make  the  most  of  GP  3.2? 


□  Select  work  products,  measures  and  measurement  results  to 
archive  and  document  what  they  will  be  used  for 

□  Define  categorization  scheme  for  work  products,  measures  and 
lessons  learned  (see  next  page) 

□  Identify  GP  3.2  activities  as  steps  in  your  processes 

□  Collection  /  contribution  points 

□  Analysis  and  categorization  points 

□  Use  of  GP  3.2  outputs 

□  Define  the  collection  or  storage  location  for  contributed  work 
products  and  measures 

□  Measure  the  effectiveness  of  your  GP  3.2  program  -  are  people 
using  the  work  products,  measures  and  lessons  learned? 
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Categorizing  work  products  and  measures 


Using  categories  similar  to  those  discussed  with 
lessons  learned,  define: 


□  Product  or  Product  Type 

□  Customer  or  Customer  Type 

□  Project  or  Project  Type 

□  Work  Product  or  Measure  Type 

□  Process  during  which  work  product  or  measure  is  used 

□  Tailoring  Criteria  or  work  product  /  task  attribute  which  suggests 
use  of  this  work  product  or  measure 


□ 
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Summary 


Take  a  thoughtful  approach  to  GP  3.2  to  ensure  long  term 
benefits  to  projects  and  the  organization 

Benefits  to  the  organization: 

□  Continuous  improvement  through  regular  incorporation  of 
best  practices  into  the  OSSP 

□  Improved  productivity  through  reuse 
Benefits  to  the  project: 

□  Clear  definition  of  what  needs  to  be  archived 

□  Quick  access  to  lessons  learned  and  relevant  examples 


31 


©  2009  Natural  SPI 


Questions 
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Contacts  and  More  Information 


□  Susan  Byrnes,  PMP 
Natural  SPI,  Inc. 

E-mail:  susan@naturalspi.com 

□  Natural  SPI’s  web  site:  www.naturalspi.com 


Real  Process 
Improvement 


Using  the 

CMMI 


Michael 

West 


Natural  SPI 


SEI Partner 
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“All  models  are  wrong  but  some  are  useful.” 


George  E.P.  Box 
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Overview 
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■  Process  Performance  Models  are  an  expected  component  of 
CMMI®  high  maturity 

-  OPP  SP  1 .5:  Establish  and  maintain  the  process-performance  models 
for  the  organization’s  set  of  standard  processes.  ^ 

■  The  concepts  of  process  performance  models  are  often 
misunderstood 

-  What  is  and  what  is  not  a  model? 

-  How  are  models  developed? 

-  When  are  models  used? 

■  Adapting  the  traditional  Goal  Question  Metric  (GQM) 
approach  to  Goal  Question  Model  can  lead  to  the 
development  of  effective,  value-added  process  performance 
models  in  an  organization. 

■  Example  from  Raytheon  Integrated  Defense  Systems  (IDS) 
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■  Process  Performance  Model  Definition  from  CMMI 

-  A  description  of  the  reiationships  among  attributes  of  a  process  and  its 
work  products  that  are  deveioped  from  historicai  process-performance 
data  and  caiibrated  using  coilected  process  and  product  measures  from 
the  project  and  that  are  used  to  predict  resuits  to  be  achieved  by 
foiiowing  a  process.  ^ 

■  CMMI  V1.2  Maturity  Level  4  and  5  appraisals  are  expected  to 
show  evidence  of  using  process  performance  models 

-  During  project  pianning/tailoring  to  compose  the  project’s  defined 
process 

-  Throughout  project  iifecycie  to  determine  if  project  wiii  achieve  its 
quality  and  process  performance  objectives 

-  May  be  used  to  support  Organizational  Innovation  and  Deployment 
(OID)  and  Causal  Analysis  and  Resolution  (CAR)  activities 
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Process  Performance  Models 

(continued) 

m  Healthy  Ingredients  of  CMMI  Process  Performance  Models  ^ 

-  Statistical,  probabilistic  or  simulation  in  nature 

-  Predict  interim  and/or  final  project  outcomes 

-  Use  controllable  factors  tied  to  sub-processes  to  conduct  the  prediction 

-  Model  the  variation  of  factors  and  understand  the  predicted  range  or 
variation  of  the  outcomes 

-  Enable  “what-if  analysis  for  project  planning,  dynamic  re-planning  and 
problem  resolution  during  project  execution 

-  Connect  “upstream”  activity  with  “downstream”  activity 

-  Enable  projects  to  achieve  mid-course  corrections  to  ensure  project 
success 
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■  Developed  by  Dr.  Victor  Basil!  working  with  NASA^ 

-  Develop  a  set  of  business  goals  and  associated  measurement  goals 

-  Generate  questions  that  define  those  goais  quantitativeiy 

-  Specify  metrics  to  be  coiiected  to  answer  those  question^ 


■  Key  is  the  trace  from  business  goal  to  metric 

-  Focus  on  what  is  most  meaningful  to  the  business 


■  Example: 

-  Goal:  Improve  the  timeliness  of  change  request  processing  from  the  project 
manager’s  viewpoint 

-  Question:  what  is  the  current  change  request  processing  speed? 

-  Metrics:  Average  cycle  time,  standard  deviation,  %  cases  outside  of  the  upper 
limit 

-  Question:  Is  the  performance  improving? 

-  Metric:  (current  average  cycle  time/Baseline  average  cycle  time)  *  100 
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Goal  Question  Model  Approach 

■  Goal 

-  Business  goals 

•  Often  not  quantitative  in  nature 

•  Example:  Improve  customer  satisfaction 

-  Quality  and  process  performance  objectives 

•  Quantitative  characterizations  decomposed  (if  necessary)  from  business 
goals^ 

•  Often  relate  to  cost,  schedule,  quality,  technical  performance 

•  Example:  Reduce  defects  in  work  products  delivered  to  customer  by  10% 
(without  additional  cost  to  customer). 

■  Questions 

-  What  factors  influence  the  achievement  of  the  goal? 

•  Example:  defect  detection  capability  during  development  and  test 

-  What  controllable  sub-processes  relate  to  those  factors? 

•  Example:  peer  reviews,  test  development  and  execution 


11/18/2009 


Page  8 


Raytheon 

Integrated  Defense  Systems 


Goal  Question  Model  Approach 

(continued) 

m  Model 

-  Identify  associated  measures 

.  Example:  defect  containment,  peer  review  measures  (preparation,  conduct, 
size,  etc.),  tools  usage  (code  static  analyzers,  simulations) 

-  Collect  data 

-  Analyze  data  for  statistical  correlations 

-  Develop  a  model  relating  factors  to  results 
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■  Goal:  Raytheon  Integrated  Defense  Systems  (IDS)  has  business  /  project 
cost  and  schedule  performance  goals  of  CPI,  SPI 

■  Question:  What  are  the  factors  influencing  IDS  projects’  ability  to  meet 
these  goals? 

-  Aggressive  program  schedules  have  increased  the  overlap  between 
life  cycle  phases 

•  Design  begins  at  risk  before  requirements  are  complete 

-  Requirements  Volatility 

•  Changing  requirements  causes  rework  for  software  and  hardware 
development 

-  Projects  have  been  unable  to  quantify  the  risks  associated  with 
these  factors. 

■  Question:  What  controllable  sub-processes  relate  to  those  factors? 

-  Requirements  management,  requirements  development,  technical 
solution 


*  Patent  pending 
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Raytheon  IDS  Example:  SLAM 

(continued) 


m  Model 

-  Identify  associated  measures 

•  CPI,  SPI  and  Requirements  Volatility  are  required  measurements  collected 
and  reported  by  every  development  project  across  Raytheon  Company. 

•  Requirements/design  overlap 

-  A  non-standard  project  measurement  collected  and  analyzed  during  the 
SLAM  development  piloting  &  deployment. 

-  SLAM  piloting  effort  worked  closely  with  a  cross-section  sampling  of  our 
IDS  development  projects  in  defining  an  objective  measurement  that  is 
easily  collected  and  readily  available. 

-  Up-front  collection  defining  dialogue  with  SLAM  pilot  project  teams 
provided  highly  valuable  analytical  and  deployment  insight. 
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Raytheon  IDS  Example:  SLAM 

(continued) 


■  Model  (continued) 

-  Collect  and  analyze  data 

•  A  mathematical  function  of  the  input  factors  was  reasonably  well  correlated  with  the 
output  responses  using  linear  regression  techniques  (with  an  adjusted  r-squared  value 
=  0.65,  p=  .000).  Additionally  collected  project  data  from  SLAM  piloting  and  deployment 
further  confirmed  the  strength  of  this  underlying  statistical  relationship. 
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Raytheon  IDS  Example:  SLAM 

(continued) 


m  Model  (continued) 

-  Crystal  Ball  was  selected  as  the  statistical  modeling  tool  of  choice  both 
because  of  its  availability  to  all  (Raytheon  has  a  Corporate  license)  and 
because  of  its  general  ease  of  use  for  project  teams. 

-  Using  the  correlated  regression  equation  and  estimates  of  mean  and 
variance  for  each  of  the  factors  (from  the  collected  data),  a  Monte 
Carlo  simulation  model  was  developed  with  an  Excel-based  User 
interface. 

-  The  SLAM  Model  User  Interface  also  includes: 

•  Crystal  Ball  download  instructions 

•  Step-by-Step  Guidance  for  Projects  on  “Running 

•  Guidance  on  how  “Interpreting  the  Results” 

•  A  Listing  of  Potential  Mitigation  Strategies 
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Raytheon  IDS  Example:  SLAM 

(continued) 

■  Model  (continued) 

-  SLAM  Model  Inputs 

•  Estimated  %  Design  Complete  at  Requirements  Release 

-  Confidence  Range  (+/-  5,  10,  or  15%) 

•  Requirements  Volatility  Estimate 

-  Enter  in  best  estimate  based  on  historical  baseline  for  product  line, 
process  tailoring,  etc. 

-  Variance  estimates  built  into  model  based  on  historical  actuals 

-  SLAM  Model  Output 

•  Projected  Software  /  Hardware  Cost  Performance  (CPI) 

-  Mean,  Standard  Deviation 

-  95%  Upper  &  Lower  Prediction  Interval  Limits 


11/18/2009 


Page  14 


Raytheon  IDS  Example:  SLAM 

(Example  1) 


Raytheon 

Integrated  Defense  Systems 


Home 


r-*  O  - 

Insert  Page  Layout  Formulas 


"■  ■0  :  V‘3  jCC't’iL  dliiTMiH 
Review  v/iew  Add-Ins  j  Crystal  Ball  I 


lA  ^  ^ 

IHIkl  Past-  I  r.- Freeze 
Define  Define  Define 
Assumption  Decision  Forecast  aiP 

Define 


iji  p  I# 

^  ^  y  Save  or  Restore  -  «=»'  ‘ ' 

Start  Stop  Reset  trp  View  Create  Extract  ^ 

gt  Run  Preferences  Charts- Report-  Data  |  A? About 


Help 

^  Resources  - 


Help 


D15 


4 

5  ^ 

8j 

jr 

J2  ; 

Jl 

14^; 
15 ; 

16  I 

17 

JI8 

19 

20 

J1 

22 

23 

24 

25 

26 
27 

30 

31 

32 

33 


%  SW/HW  Detail  Design  Complete  at  Requirements  Release 


Simulation  complete 

0 

T  otal  trials:  1 ,000 

1,000 

tM  lir 


^lAM  MnAJ  ^  CB  Downtoad  Instructions  ^  Running  SLAM 


Forecast:  CPI 


1%  Design  Complete  at  Requirements  Release 

1  55  1 

1  Range  (+/-%) 

1  io  1 

Requirements  Volatility 

1  Requirements  Volatility  Estimate(%) 

1  50  1 

Predicted  SW/HW  CPI 

Mean 

0.816 

Standard  Deviation 

0.017 

95%  Prediction  Interval  Lower  Limit 

0.784 

95%  Prediction  Interval  Upper  Limit 

0.848 

Edir  ViP'A  Fornr^r  Preterences 
1 ,000  T  rials  Frequency  Vie w 


1 ,000  Displayed 


CPI 


o.oo^ - ' 

0.76  0.77  0.79  0.80  0.82  0.83  0.85  0.87  0.8 


^  l-Infinity 


Certainly:  1 100. 00  X 


I  Infinity 


Strategies  Revsion  History  ,  ^ 


r 


Ready 


^  goal  question  met. . .  |  ©  Microsoft  Offic. . . 


(  >Re:  Cost  Model ... 


I  Internet  Explorer  -  jfe^GQM 


Crystal 


8/2009  Page  15 


Raytheon  IDS  Example:  SLAM 

(Example  2) 


Raytheon 

Integrated  Defense  Systems 


d 


Home  Insert  Page  Layout  Formulas  Data  Review  View  Add-Ins  ;  Crystal  Ball  | 


1  I 

2_\ 

3  ^ 

4  i 

A  i 

7  I 

8  ~ 

10  I 

Jl! 

JI2  I 

13  i 

14  i 
1^1 
16  I 
17 ; 

18  i 

19  I 

20  I 

21  i 
22 : 
23  I 
24 ; 

26 ; 

27  ' 

28  i 

29  I 

30  ! 

31  i 

32  i 

33  j 


%  SW/HW  Detail  Design  Complete  at  Requirements  Release 


1%  Design  Complete  at  Requirements  Release 

1  30  1 

1  Range  (+/-%) 

1  5  1 

Requirements  Volatility 

1  Requirements  Volatility  Estimate(%) 

1  25  1 

Predicted  SW/HW  CPI 

Mean 

0.980 

Standard  Deviation 

0.006 

95%  Prediction  Interval  Lower  Limit 

0.968 

95%  Prediction  Interval  Upper  Limit 

0.992 

Control  Panel 


Run  Analyze  Help 


Simulation  complete 

0 

T  olal  trials:  1 ,000 

1,000 

^  I  ^  H  ^  11^ 


CB  Download  Instructions  ^  Running  SLAM 


thel 


Strategies  ^  Revision  History  , 


Ready 


P  goal  question  met. . .  ^  Microsoft  Offic. . .  jJJ*  >  Re:  Cost  Model  ^ 


t  Explorer 


5LAMV6.0.3.xls  [.. 


-  -r-  X 


A 

Define 

Assumption 

Define 

Decision 

ib 

Define 

Forecast 

45Copy 

^  Clear 

Select  - 

i  f-  Ff  =  -.i. 

Cell  Prefs 

start  "top 

Reset 

^  Q  Save  or  Restore 

Run  Preferences 

l^  l 

View  Create  Extract 
Charts-'  Report’  Data 

#Help 

Resources  ’ 

About 

Define 

Run 

Analiie 

Help 

D28 

jSr 

1  A  ; 

B 

c 

1 

D  'T 

1/18/2009  Page  16 


Raytheon 

Integrated  Defense  Systems 


Raytheon  IDS  Example:  SLAM 

(continued) 

u  The  System  Lifecycle  Analysis  Model  (SLAM)  was  developed  and 
deployed  at  Raytheon  Integrated  Defense  Systems  In  order  to 
quantitatively  assess  the  cost  performance  risk  associated  with 
requirements  volatility  and  requirements  /  design  lifecycle  overlap. 

■  SLAM  has  been  used  to  identify  risks  during  early  planning  as  part  of 
proposal  activity 

■  SLAM  Is  used  by  Integrated  project  teams  made  up  of  Systems,  Software, 
Hardware  and  Quality  Engineering  during  project  planning  and  execution 

-  Quantifies  risk 

-  Enables  composition  of  project’s  defined  process 

-  Manages  against  quality  and  process  performance  objectives 

■  The  Engineering  Process  Group  has  used  SLAM  to  estimate  benefits  of 
process  improvement  proposals  and  to  measure  changes  in  performance 
due  to  process  improvements. 
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•  SLAM  was  built  to  aid  IDS  programs  in  their  ability  to  achieve 
objectives.  The  users  of  the  model  felt  it  recognized  their  issues, 
validated  their  program  concerns,  and  would  help  them  support 
resolution  of  these  issues. 

•  Keeping  the  model  simple  and  fast  to  use  was  seen  as  a  plus  by  users. 
No  training  beyond  a  short  demo  was  required. 

•  Reviewing  data  with  project  people  revealed  insights  that  led  model  to 
be  developed  in  a  different  manner  than  planned  and  contributed  to 
buy-in  from  users.  The  iteration  back  to  the  data  providers  was 
invaluable  to  the  developers. 

•  Demonstrating  the  model  with  Engineering  managment  helped  them 
understand  the  tool,  provided  commitment  to  pilot  and  deploy. 

•  Let  the  data  lead  you  to  a  solution.  Original  SLAM  concept  was 
different  than  what  was  actually  created. 

•  Start  small,  get  buy-in,  and  build  from  there.  Expectation  is  for  SLAM  to 
expand  with  more  granularity  across  the  lifecycle. 
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■  The  Goal  Question  Metric  approach  emphasizes  defining 
measurements  in  a  top  down  fashion 

-  Measure  what  is  most  meaningful  to  the  business  or  a  project 

-  This  is  consistent  with  the  CMMI  expectations 

•  Measurement  objectives  are  derived  from  information  needs  (MA) 

•  Establish  quality  and  process  performance  objectives  (OPP,  QPM) 

■  Using  the  same  top  down  approach  with  process 
performance  modeling  produces  value-added,  effective 
results 

-  Model  what  is  most  meaningful  to  the  business  or  project  focuses 
resources  where  the  value  is  greatest 

-  Leads  to  acceptance  and  use  of  the  models  at  various  levels  of  the 
organization 

-  Helps  the  business  or  project  achieve  its  objectives 
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WHAT  WE  ARE  GOING  TO  DO  and  WHY 


We  are  going  to  evaluate  a  program  In  terms 
of  the  schedule,  costs,  fee  structure  and 
estimated  returns. 

We  want  to  know  if  we  are  going  to  make 
money. 


ACKNOWLEDGEMENT:  The  approach  within  this  presentation  has  been 
inspired  by  the  teachings  of  Dr.  Sam  Savage  and  my  demonstration  will  use 
Risk  Solver™  software  available  through  Frontline  Systems;  other  useful 
software  is  ©Risk  from  Palisade  Corporation  and  Oracle  Crystal  Ball.  2011 


HOW  WE  ARE  GOING  TO  DO  IT  and  WHY 


First,  using  a  deterministic  approach  for  estimating 
factors,  using  expected  or  desired  point  value 
assumptions. 

Second,  using  a  stochastic  approach  for  estimating 
factors,  using  a  variety  of  stochastic  assumptions. 

We  want  to  know  if  we  are  going  to  make  money;  AND 
how  sensitive  our  outcome  may  be  based  on  our 
assumptions  and  the  unknown  uncertainties. 


3  of  11 


OVERVIEW  OF  THE  PROGRAM 


The  Program:  Requirements 

Provide  a  recommendation  with  supporting  analysis 
regarding  whether  or  not  a  proposed  organizational 
change  should  be  implemented. 

We  have  three  interim  deliverables  due  at  150,  180,  210 
days  respectively  with  a  final  deliverable  due  in  360 
days. 
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OVERVIEW  OF  THE  PROGRAM  CONTINUED _ 

The  Program:  Fee  Determination 

The  first  deliverable  is  due  in  150  days  and  is  worth  $100K 

The  second  deliverable  is  due  in  180  days  and  is  worth  $150K 

The  third  deliverable  is  due  in  210  days  and  is  worth  $200K 

The  final  deliverable  is  due  in  360  days  and  is  worth  $550K 

Each  interim  deliverable  is  subject  to  a  10%  bonus  if  more 
than  20  days  early  and  a  10%  penalty  if  more  than  10  days 
late. 

The  final  deliverable  is  subject  to  a  10%  bonus  if  more  than 
60  days  early;  a  5%  bonus  if  more  than  30  days  early;  a  10% 
penalty  if  more  than  20  days  late;  and  a  25%  penalty  if 
more  than  40  days  late. 
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SOME  QUESTIONS... _ 

How  long  will  It  take? 

How  much  of  the  potential  fee  will  we  earn? 
How  much  will  it  cost? 

Ultimately  -  How  much  money  will  we  make? 
And...are  you  sure? 
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HERE  ARE  THE  AAAIN  TASKS  AND 
DELIVERABLES... 


Here  they  are  again,  identified  as  Main  Tasks  (MT)  and  Deliverables  (D)  v/ith 
estimates  for  times  and  costs;  base  fee  amounts  for  each  deliverable  as  well 
as  the  critical  path  identified  for  each  deliverable  and  overall... 
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DETERMINISTIC  EVALUATION... _ 

The  expected  time  and  anticipated  fee  for  each  deliverable 
and  expected  overall  program  cost  are  therefore... 

D1  -  115  Days  =  $100K  (No  bonus  or  penalty) 

D2  -  170  Days  =  $150K  (No  bonus  or  penalty) 

D3  -  200  Days  =  $200K  (No  bonus  or  penalty) 

D4  -  350  Days  =  $550K  INo  bonus  or  penalty) 

Total  Fee  =  $1M  (No  bonus  or  penalty) 

Total  cost  =  sum  of  costs  for  all  tasks  and 
deliverables  =  $880K 
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We  will  finish  on  time  for  each  deliverable  and 
overall,  we  will  earn  no  bonuses  but  incur  no 
penalties  and  we  will  make  $120,000 
(approx.  13.6%  return) 

We  mapped  all  process  steps  and  documented  all 
assumptions 

What  could  go  wrong? 


Shift  to  Demonstration 
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NORmROR  GRUMMAN 


Lessons  Learned 
Piloting  the 
CMMI  for  Services 

CMMI  Technology  Conference 
November  16-19,  2009 


Diane  Mizukami-Williams 
Northrop  Grumman  Corporation 


NORTHROP  GRUMMAN 


Northrop  Grumman  Information  Systems  (IS)  Sector 


I S  Sector 

•  $10  billion  in  sales  in  2008 

•  7,000  contracts 

•  33,000  employees 

Products  and  Services 


Mission  support 
Cybersecurity 

Commancd,  control,  ancd  communications 
Enterprise  applications 
IT  &  network  infrastructure 
Management  &  engineering  services 
Intelligence,  surveillance,  &  reconnaissance 


»ecce: 


2 


I S  as  a  CMMI  for  Services  Early  Adopter 


NORTHROP  GRUMMAN 


•  I S  has  a  history  of  successful  CMMI  adoption 

-  One  of  the  first  large  organization  adopters 

-  Over  80  organizations  (over  250  projects)  appraised  at  Level  3  or  higher 

•  I S  was  very  interested  in  applying  our  successes  to  services 

•  Strong  IS  involvement  in  developing  the  CMMI  for  Services  model 

-  Hal  Wilson  -  CMMI  Steering  Group  advocate  for  developing  the  model 

-  Craig  Hollenbach  -  Model  Project  Manager 

-  Brandon  Buteau  -  Model  Architect 

-  Roy  Porter  -  One  of  the  model  authors 

•  Made  sense  for  I S  to  be  an  early  adopter 

•  IS  completed  a  successful  Level  3  SCAMPI  A  in  October  2009 

-  Led  by  Pat  OToole  and  3  lead  appraisers  (J  ohn  Clouet,  Ron  Ulrich,  Ravi  Khetan) 
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SCAMPI  A  Projects 


NORTHROP  GRUMMAN 


•  Started  with  4  pilot  projects 

•  Positives 

-  All  previously  appraised  at  CMMI  Level  3  or  5 

-  3  projects  were  service-only,  1  was  software/ hardware/ service 

-  Felt  adopting  the  model  would  improve  their  processes 

•  Negatives 

-  Projects  were  apprehensive  about  the  newness  of  CMMI  -SVC 

-  Wanted  assurance  that  IS  experts  would  assist  them  in 
understanding  the  model  and  helping  with  improvements  and 
artifacts 

•  Business  reasons  eventually  reduced  the  appraisal  to  1  project 

-  IS  and  the  project  could  still  benefit 
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7  CMMI  for  Services  Unique  Process  Areas 


NORTHROP  GRUMMAN 


L2 


L3) 


ervice  Delivery 

Deliver  services  per 
service  agreements 


Capacity  and 
Availability 
Management 

Effective  performance 
and  resources  are 
used  effectively 


L3 


Service 
Continuity 

Continuity  of  services 
during  and  following 
significant  disruptions 


L3 


Service  System 
Transition 

Deploy  a  new  or 
changed  service 
system 


I  ncident 
Resolution  and 
Prevention 

Resolution  and 
prevention  of  service 
incidents. 


L3 


Service  System 
Development 
OPTI ONAL 

Design,  develop, 
integrate,  verify,  etc 
the  service  system. 


L3 


Strategic  Servic 
Management 

standard  services  per 
strategic  needs  and 
plans 


L3 


Note:  Also  1  new  practice  in  OPD  and  PP. 
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Easy  to  More  Painful  Process  Areas 


NORTHROP  GRUMMAN 


Strategic 
Service 
Management 


Service 

Continuity 


Capacity  & 
Availability 
Management 


Service 
System 
Development 


Service 

System 

Transition 


I  ncident 
Resolution  & 
Prevention 


Presentation  will  cover  what  we  learned 
from  the  easiest  (Service  Delivery) 


Service 

Delivery 


through  the  most  painful  (Strategic 
Service  Management)  process  areas 


6 


1  Service  Delivery 


NOKTHROP  GRUMMAN 


Positives 


Projects  naturally  implennented  service  delivery 

-  Projects  had  service  agreennents 

-  Projects  prepared  for  service  delivery 

-  Projects  delivered  services 


Slight  Difficulty 


Analyzing  existing  agreennents  and  service  data  (SP  1.1) 

-  Projects  may  or  may  not  do  this,  and  even  if  they  did,  it  may 
not  be  documented 
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2  I  ncident  Resolution  and  Prevention 


NOKTHROP  GRUMMAN 


Positives 


Model  improves  trouble  tickets 

-  Projects  added  more  fields  to  capture  nnore  data  for  trending 

-  Encouraged  capturing  information,  i.e.,  write  it  down 


Slight  Difficulty 


Workarounds  (SP  2.3) 

-  Workaround  repository  is  not  required,  but  the  model 
mentions  it,  and  projects  generally  do  not  have  one 

-  Workaround  used  is  not  always  documented 


Confusion 


Incidents  (Goal  2)  versus  problems  (Goal  3)  not  clear 

-  Not  all  “incidents"  are  a  "problem".  Someone  might  report  an 
incident,  'The  computer  is  broken".  Your  response,  'You 
didn't  turn  it  on".  It's  not  a  "problem"  unless  it  happens  a  lot. 

-  Model  team  is  correcting  the  confusion  in  VI. 3 
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3  Service  System  T ransition 


NOKTHROP  GRUMMAN 


Positives 


Confusion 


Model  adds  nnore  discipline  for  transitions 

-  Encourages  better  planning  for  transitions 

-  Ensures  impacts  are  known  and  impacts  are  monitored 

-  Ensures  people  are  prepared  for  changes 

-  Stops  dump  and  run  attitude,  “Here  you  go,...  good  luck" 

Transition  tends  to  be  infornnal 

-  T ransition  plans  may  or  may  not  exist 

-  Monitoring  impacts  tends  to  be  informal,  “Hey,  how's  it  going?" 

-  More  difficult  to  gather  evidence 

None 
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4  Service  System  Development 


NORTHROP  GRUMMAN 


Ensures  all  life-cycle  activities  are  addressed 

Projects  are  very  happy  to  use  a  nnodel  that  fits  their  work 
(CMMI  for  Development  more  painful) 

Optional  (should  use  for  complex  service  systems) 

Software/ hardware/ service  projects  miss  services 

-  Have  plenty  of  evidence,  but  very  little  for  services 

-  For  example,  GP  2.8  status  reports  only  address  the  software  / 
hardware  product,  but  not  the  service  system 

-  Service-only  projects  are  much  easier  to  work  with 

I  nclude  the  optional  process  area  or  not  ??? 

I  n  VI. 3,  SSD  will  likely  NOT  be  an  option.  Projects  must 
provide  rationale  why  it  is  N/A  like  SAM. 


5  Capacity  and  Availability  Managennent 


NOKTHROP  GRUMMAN 


Confusion 


Ensures  projects  monitor  these  critical  items 

Helps  formalize  both  capacity  and  availability 

Ensures  measures  are  collected  and  analyzed,  which  is  good 

Availability  and/or  capacity  not  done 

If  done,  not  done  formally 

Only  done  well  if  a  contract  requirement 

Should  be  at  the  service  system  level,  not  component  level, 
although  key  components  should  do  it 

Service  system  representation  (SP  1.3)  does  not  have  to  be 
graphical,  but  must  provide  useful  information  (Buteau) 
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6  Service  Continuity 


NOKTHROP  GRUMMAN 


Positives 


Projects  generally  do  not  think  of  continuity  until  a  major 
disruption  occurs 

Puts  things  in  place  BEFORE  a  major  disruption  occurs 
Brings  structure  to  planning  and  implementation 


Confusion 


Lack  of  Service  Continuity  Plans  (SP  2.1) 

-  Assume  they  will  not  have  sufficient  plans 

-  Created  a  detailed  53  page  Service  Continuity  Plan  Template 

-  Template  helped  projects  tremendously 

Verify  and  validate  the  Service  Continuity  Plan  (SP  3.2) 

-  People  are  not  used  to  testing  and  validating  a  "plan" 

-  Educated  the  project  using  the  template 

-  Key  services  and  essential  functions  and  resources  in  the  plan 
should  be  verified  and  validated  (Buteau) 
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7  Strategic  Service  Management 


NORTHROP  GRUMMAN 


Last  but  not  least,... 
Strategic  Service 
Management 
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7  Strategic  Service  Management  ( 1  of  2) 


NOKTHROP  GRUMMAN 


Positives 


Ensures  the  long  term  health  of  the  service 

Evolves  the  service  per  market  and  customer  needs  so 
service  does  not  stagnate  over  time 

Makes  it  very  clear  what  services  are  provided 


Very  Difficult 


Properties  of  standard  services  and  service  levels  (SP  2.1) 

-  Model  fits  cell  phone  companies  with  similar  services 

-  Much  more  difficult  with  companies  as  diverse  as  Northrop 
Grumman  (Red  Cross  blood  bank  project,  anti-terrorist  FBI  project. 
Internal  Revenue  System  (IRS)  project,  etc.) 

-  Pick  the  level  in  the  organization  where  things  become  more 
common 

-  Used  project  evidence.  Project  had  a  "Chinese  Menu"  where  you 
order  this  for  your  site,  and  that  for  your  site,  etc. 
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7  Strategic  Service  Management  (2  of  2) 


NOKTHROP  GRUMMAN 


Confusion 


STSM  is  project  or  organization? 

-  I  mmediate  reaction  was  STSM  was  a  project- level  process  area 

-  STSM  is  not  in  the  Process  Management  category  like  OPD,  OPF, 
etc. 

-  According  to  the  authors,  it  was  intended  to  be  organizational, 
similar  to  OPD,  OPF,  etc. 

STSM  is  not  like  OPD,  OPF,  etc. 

-  OPD,  OPF,  etc.  evidence  works  whether  there  is  1,  2,  or  100 
projects 

-  I  n  STSM,  switching  to  1  project  changes  the  evidence 

-  Populated  PHD  with  Sector,  Division,  Department,  and  project 
evidence 

-  Appraisal  team  called  a  1  hour  telecon  for  STSM 

-  Debated  on  who  should  do  this.  Sector?  Division?  Business 
Unit?  Departnnent?  Project? 

-  Model  authors  stated  practices  could  be  done  at  one  or  more 
levels,  ...  it  depends 
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Miscellaneous 


NORTHROP  GRUMMAN 


•  What  is  a  service  project 

-  A  “project"  covers  the  scope  of  one  service  agreement,  which  may  contain 
several  services  (Buteau) 

-  One  appraisal  team  member  felt  each  service  within  a  project  could  be 
treated  as  a  separate  “project"  and  should  do  every  practice 

•  Enhance  training 

-  Expand  appraisal  team  member  training 

-  Appraisal  team  appraised  development  projects  for  so  long,  they  may  not 
be  able  to  shift  their  thinking  to  services 

•  Typical  “gap  analysis"  approach  wont  work 

-  Address  the  7  new  service-specific  PAs  and  the  1  additional  practice  in  PP 
and  OPD  and  I 'm  done, . . .  WRONG 

-  Half  way  through,  realized  OPD,  OPF,  etc.  only  contained  evidence  for 
systems/ hardware/ software  but  nothing  for  services.  Reworked  OPD,  OPF, 
etc.  to  add  service- specific  evidence. 
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Summary 


NORTHROP  GRUMMAN 


•  Dont  assume  if  you  address  the  7  new  service- specific  Process 
Areas  (and  1  PP  and  OPD  practice),  you're  done 

•  Beware  of  Strategic  Service  Management 

•  Overall,  transitioning  to  the  new  CMMI  for  Services  model  was  a 
great  idea 

•  Recommend  using  the  model,  ...  I  like  it! 


Contact  I  nformation: 

Diane  Mizukanni  (Williams) 

Diane.  Mizukami(a)ngc.com 
310-921-1939 

Northrop  Grumman  I  nformation  Systems 
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Work  ON  Your  Engineering 
Business,  Not  jN  It! 


NDIA  CMMI  Technology  Conference 
November  17^'’,  2009 
Rolf  W.  Reitzig 

cognenceinc 

Improving  Software  Economics 

©2009-  cognence,  inc. 


What  Is  Your  Competitive  Advantage? 


•  Technology? 

•  People? 

•  Quality? 

•  Cost? 

HOW  you  develop  your  products  strongly  determines  WHAT 
competitive  advantages  you  are  able  to  obtain  and  maintain 
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How  Is  Your  Company  Run? 


•  Day-to-Day? 

•  Quarter-to-Quarter? 

•  Year-to-Year? 

•  Multi-Year? 

What  activities  does  management  engage  in  that  overiay  the 
activities  overwheiming  empioyees  on  a  day-to-day  basis?  Or, 
is  management  consumed  by  the  day-to-day  as  well? 

What  insight  does  management  have  into  the  business  model 
that’s  being  empioyed,  and  how  to  improve  and  refine  it? 
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Successful  Businesses... 


•  Run  operations  as  if  they  were  a  franchise 


-  Every  business  process  is  standardized 

-  Average  employees  can  easily  be  successful 
by  following  the  processes  as  outlined 

-  Well  executed  processes  are  scaled  and 
leveraged  across  the  organization 

•  For  software  organizations,  “franchising” 
processes  can  result  in  a  50%  or  more 
increase  in  productivity 
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Jim  Collins’  Good  to  Great 


•  Good-to-great  companies  focus 
equally  on  what  to  do,  what 
not  to  do,  and  what  to  stop 
doing 

•  Technology-driven  change  had 
virtually  nothing  to  do  with 
igniting  a  transformation. 
Technology  can  help 
accelerate,  but  doesn’t  cause 

change.  Technology 
influences  typically  come  last, 
not  first. 
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#1  BESTSELLER 
TKREE  MILLION  COMES  SOLD 


Slides 


Key  Concept  From  Good  to  Great 


“Good-to-great  companies  built  a  consistent 
system  with  clear  constraints,  but  they  also 
gave  people  the  freedom  and  responsibility 
within  the  framework  of  that  system.  They 
hired  self-disciplined  people  who  didn’t  need  to 
be  managed,  and  then  managed  the  system, 
not  the  people.” 
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other  Good  to  Great  Thoughts 


•  "What  are  the  brutal  facts?  We've  got  to  get  a  grip  on 
the  facts,  what  are  the  trends,  what  are  the  trendlines, 
how  bad  is  it?  Get  a  grip  on  the  facts." 

•  "How  does  a  culture  of  mediocrity  take  hold?  The 
signature  of  mediocrity  is  chronic  inconsistency” 

•  "What  you  can  measure  you  can  target.  And  what  you 
can  target  you  can  accomplish.” 

•  "Don't  look  for  silver  bullets.  Pick  a  lead  bullet  and 
polish  it  so  it  becomes  silver" 
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Key  Franchising  Concepts 


•  Great  businesses  are  not  built  by  extraordinary  people, 
but  by  ordinary  people  doing  extraordinary  things 

•  To  achieve  this,  a  system  is  absolutely  essential  -  it 
becomes  the  tools  people  use  to  increase  productivity, 
to  get  the  job  done  in  a  way  that  differentiates 

•  A  franchise  is  simply  your  unique  wav  of  doing 
business  -  vour  system 

•  If  you  haven’t  orchestrated  your  business,  you  don’t 
own  it! 


Source:  The  e-Myth  Revisited,  Michael  E.  Gerber,  HarperCollins  Publishers,  1995 
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Management’s  Role 


•  It’s  management’s  job  to  develop  systems  and  tools  and 
teach  employees  how  to  use  them 

•  Its  the  employee’s  job  to  use  the  systems  and  tools  and 
to  recommend  improvements  based  on  their  experience 
with  them 

•  Management  makes  sure  employees  understand  the 
idea  behind  the  work  they  are  being  asked  to  do 

•  Avoid  “Management  by  Abdication”! 


Source:  The  e-Myth  Revisited,  Michael  E.  Gerber,  HarperCollins  Publishers,  1995 
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The  Capability  Maturity  Model  Integration 


•  The  CMMI  is  a  framework  that  describes  the  key  elements  of  an 
effective  systems  and  software  process,  and  provides  for  an 
evolutionary  improvement  path  from  an  ad  hoc,  immature 
process  to  a  mature,  disciplined  one. 

•  The  CMMI  guides  engineering  organizations  that  want  to  gain 
control  of  their  processes  for  developing  and  maintaining 
systems  and  software  and  to  evolve  toward  a  culture  of  software 
engineering  and  management  excellence. 

•  The  intent  of  the  CMMI  is  to  install  a  process  infrastructure  that 
supports  standardization,  scalability,  continuous  re-evaluation, 
and  improvement  -  in  other  words,  an  engineering  system 
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Support 


CMMI  Engineering  Business  Model  Philosophy 


Process  Management 

OPF 

OPD 

OT 

Integrated  Project  Management 

Project 

Project  Monitoring  and  Control 

Q- 

Planning 

Risk  Management 

Supplier  Agreement  Management 

Engineering 

REQM  RD  TS  PI  VER  VAL 

_ Configuration  Management 

_ Measurement  and  Analysis _ 

Process  and  Product  Quality  Assurance 
Decision  Analysis  and  Resolution _ 
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Generic  Practices 


2.1 

Establish  an  Organizational  Policy 

2.2 

Plan  the  Process 

2.3 

Provide  Resources 

2.4 

Assign  Responsibility 

2.5 

Train  People 

2.6 

Manage  Configurations 

2.7 

Identify  and  Involve  Relevant  Stakeholders 

2.8 

Monitor  and  Control  the  Process 

2.9 

Objectively  Evaluate  Adherence 

2.10 

Review  Status  with  Higher  Level  Management 

3.1 

Establish  a  Defined  Process 

3.2 

Collect  Improvement  Information 

11/18/2009 
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Slide  12 

Examples  of  Working  IN  versus  ON  Your  Business  - 1 


IN 

ON 

Reactina  to  oroiect  oroblems 

Institutinq  a  cross-proiect 
measurement  and  periodic 

review  prooram  that 

after  they  occur 

illuminates  and  addresses 

Becomina  oood  at  respondina 

potential  issues  before  they 

occur 

to  customer  complaints, 

instead  of  eliminating  them 

Reviewing  how  projects  are 

J 

accomplishing  their  work,  and 
that  they  are  following  the 
y^anization’s  expectations 
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Examples  of  Working  IN  versus  ON  Your  Business  -  2 


Letting  teams  approach 
projects  however  they’d  like, 
and  bringing  in  whatever  tools 
they  want 


J 


ON 


Instituting  a  consistent 
engineering  process,  and 
constantly  measuring  and 
refining  it  based  on  facts 

Ouantitativeiv  evaluating  and 

implementing  new 
technologies/tools  in  a 
disciplined  fashion 

J 
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Examples  of  Working  IN  versus  ON  Your  Business  -  3 


r  ^ 

IN 

ON 

Allowing  key  project  decisions 
to  be  made  by  the  political/ 
influential  power  of  certain 
project  team  members 

Requiring  DAR  to  be  used  in  a 
light-weight,  but  quantitative 
fashion  to  remove  individual 
and  political  influence  out  of  the 
process 

J 
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Example:  65%  More  Productivity 


Average 


Leading 


Requirements  I  Design 
4.5  Months  I  6.6  Months 


Requirements  I  Design 
4.0  Months  I  6.0  Months 


Code 

8.8  Months 


Test 

9.4  Months 


Rework 


Leading  organizations 
spend  20%  less  on  software 
development  effort  and 
schedule,  and  50%  less  on 
testing  costs! 


Leading  organizations 
deliver  80%  fewer  defects  to 
production,  resulting  in 
significantly  less  rework 
and  45%  more  productivity! 


20% 


45% 


COMPARE:  Trailing  organizations  spend  30%  of  project  resources  on  testing,  leading  organizations  spend  15% 
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Leverage 


Standardization  has  a  leverage  effect 
beyond  just  quality  and  productivity 
increases 

Improved 

Current  Capability 

Capability 
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Higher  Revenue  and 
Profits  through 
More  Marketabie 
Products 
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The  Economics  of  CMMI 


STRENGTH  TIIKUEGii  INOLSTRV  &  TECHNOLOGV 


CMMI  is  an  investment 

•  Are  you  obtaining  the  returns  you  should? 

•  Is  performance  improving? 

•  Do  benefits  outweigh  the  costs? 

•  Or  just  an  added  cost  of  doing  business? 

Value  often  stems  from  business  choices 

•  Organizational  objectives 

•  Performance  goals 

•  Implementation  strategies 


These  choices  are  under  an  organization’s  control 

•  Utilize  effective  strategies  and  mechanisms  to  achieve  improved 
business  performance  and  cost  efficiencies 
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The  Effective  Use  of  CMMI® 

~  NDIA  Position  Paper 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLUGV 


Summary  of  NDIA  industry  position  statements  for  obtaining  best  value 
from  CMMI  investments: 


1 .  Good  processes  increase  the  likelihood  of  achieving  successful  project 
performance 

2.  CMMI  Is  a  model,  not  a  standard  -  adapt  CMMI  to  your  business  environment, 
resources,  and  objectives 

3.  Focus  on  business  improvement  objectives  -  a  primary  emphasis  on  achieving 
levels  may  not  achieve  significant  benefits  and  may  increase  rather  than 
decrease  costs 

4.  High  maturity  is  a  business  case  -  Justify  the  investment;  many  organizations 
find  business  value  In  Improving  processes  even  at  lower  CMMI  maturity  levels 

5.  Maturity  level  ratings  are  not  alone  a  predictor  of  project  performance  -  many 
other  factors  can  be  significant  contributors 

6.  Don’t  specify  maturity  levels  In  acquisitions  -  use  CMMI  to  probe  supplier 
capability  and  process  execution  risks 

7.  Greatest  benefits  of  appraisals  are  from  improvements,  not  evidence  or  ratings  - 
disproportionate  effort  on  appraisal  preparation  risk  can  diminish  business 
returns 


•‘The  Effective  Use  of  CMMI®”,  NDIA  Systems  Engineering  Division,  June  2009. 

http://www.ndia.orq/Divisions/Divisions/SvstemsEnqineerinq/Documents/CMMI%20Workinq%20Group/CMMI%20NDIA%20position%20statement  final  .pdf 
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The  Economics  of  CMMI 


STRENGTH  TIIKUEGii  INULSTRV  &  TECHNOLOGV 


Overview: 

•  Developed  by  NDIA  CMMI  Working  Group 

•  Guidance  by  industry,  and  for  industry,  on 
achieving  business  value  through  CMMI 

•  Suggested  CMMI  strategies  and  mechanisms, 
intended  to  be  tailored  much  like  the  model  itself 

Content: 

1.  Guidance  on  achieving  business  performance 
improvement  through  economical  use  of  CMMI 

2.  Guidance  on  effective  CMMI  implementations  to 
address  common  business  issues 

Objectives: 

•  Provoke  thoughtful  dialog  on  the  effective  use  of  CMMI 

•  Influence  the  mindset  of  CMMI  business  value  -  focus  on  improvement 

•  Help  raise  expectations  across  industry  for  results  achieved  through  CMMI 


nviim 


The  Econoidci  of  CMXO 


The  Economics  of  CMMI* 
NDL\  Systems  Engineering  Division 
CMMI  Working  Group 
Version  I.O 
October  15, 2009 
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The  Economics  of  CMMI  - 

Targeting  CMMI  Decision-Makers 


STRENGTH  T1IKUEGU INULSTRV  &  TECHNOLOGV 


Section 

Topics 

Economicai  Business 
Appiication  of  CMMi 
(Executives) 

•  Support  of  Business  Goals  and  Strategy 

•  Organizational  Leadership 

•  Improvement  Velocity 

•  Making  Performance  Improvement  Intrinsic  to  the  Job 

Economicai 

impiementation  of  CMMi 
(Implementers) 

•  Use  CMMI  as  an  Integrating  Framework 

•  Develop  and  Deploy  Processes  Effectively 

•  Tailor  CMMI  Implementation  Appropriately 

•  Implement  CMMI  in  a  Practical  Way 

•  Make  an  Informed  Decision  on  High  Maturity 

•  Conduct  Appraisals  Economically 
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Economical  Business  Application 
ofCMMKPartD 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLUGV 


CMMI  business  value  depends  on  a  foundation  of  underlying  principles: 


First  Principles  of  CMMI  Adoption 

Potential  Impact  When  Not  Adopted 

CMMI-based  improvement  efforts  must 
align  with  and  support  defined  business 
goals. 

CMMI  investments  do  not  affect  business  performance; 
process  improvements  which  are  not  really  improvements 
have  detrimental  effects. 

Organizational  leadership  must  be  actively 
involved  and  visibly  committed  to  the 
improvement  effort. 

Improvements  are  not  substantial  or  lasting,  due  to  lack  of 
organizational  commitment  and  resources.  Missed 
opportunities  to  improve  the  business. 

Manage  process  improvement  velocity. 

The  rate  at  which  processes  are  improved 
must  respond  to  the  needs  of  the 
business. 

Massive  simultaneous  change  overwhelms  an  organization 
and  results  in  loss  of  focus  on  high  priority  improvement 
targets.  Improvements  are  not  realized  in  a  reasonable 
time  frame,  which  reduces  the  return  on  investment. 

Continuous  performance  improvement 
must  be  an  intrinsic  part  of  the  iob  -  not 
secondary  to  it. 

Workforce  not  engaged  in  improvement  initiatives.  Waste 
due  to  inefficiencies  and  organizational  resistance  to 
change.  Premature  abandonment  based  upon  failures 
leaving  a  worsened  condition  in  the  aftermath. 

The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 


6 


Support  of  Business  Goals  and  Strategy 

TiaUf  kt  fill  llWItl  sn'OV  Jt  TIi'f'lfiLIII  /WIV 


CMMI  is  for  improvement  with  a  purpose 

•  Fit  CMMI  to  the  business  objectives,  not  vice  versa 

-  Improving  cycle  time,  productivity,  quality, 
cost  efficiency,  customer  satisfaction,  etc. 

•  CMMI  is  a  means  to  an  end  -  not  the  objective  itself 


Prioritize  improvements  where  business 
performance  needs  are  greatest 

•  What  business  issues  are  being  faced? 

•  How  can  CMMI  help  address  them? 


CMMI  Implementation 


Pursue  business  value  and  improved 
performance 

•  Disproportionate  emphasis  on  maturity  levels  can 
lead  to  a  compliance-focused  approach  with 
burdensome  processes  at  increased  cost 


Economical  Business  Application 

>  Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Organizational  Leadership 


STRENGTH  TIIKUEGil  INOLSTRV  &  TECHNOLOGV 


Prominent  executive  sponsorship  of  CMMI 

•  Management  commitment  is  crucial 

•  Set  and  communicate  the  strategic  vision 

•  Provide  adequate  resources  (staff,  funding,  tools) 

•  Model  and  reinforce  desired  behaviors 

Hold  peopie  accountabie  for  improvement  progress 

•  Set  objectives 

•  Get  the  organization  involved 

•  Recognize  and  reward  achievements 


Understand  and  communicate  CMMI  commitment 

•  Set  the  tone  on  why  CMMI  is  important 

•  The  workforce  will  follow  cues  from  management 


Economical  Business  Application 

Support  Business  Goals/Strategy 
>  Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 


The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 


8 


Improvement  Velocity 


STRENGTH  TIIKUEGil  INDLSTRV  &  TECHNOUXiV 


Manage  process  changes  at  the  rate  needed  to 
support  the  business 

•  What  changes  are  needed,  in  what  timeframe? 

Plan  for  change  at  the  organizational  level 

•  Factors  influencing  the  ability  to  absorb  change 

-  Relationships  of  processes  with  performance 

-  Current  state  of  processes  and  leadership 

-  Project  profiles  (size,  complexity,  domain,  etc.) 

-  Improvement  strategies  and  methods 

•  Prioritize  improvements  where  most  needed 

Manage  process  improvement  like  a  project 

•  Apply  the  same  rigor  as  for  any  key  project 

•  Led  by  a  capable  project  manager 

•  Org  charts,  with  defined  roles  and  responsibilities 

•  Budget,  schedule  milestones,  project  reviews 

•  Engage  the  appropriate  stakeholders 


Economical  Business  Application 

Support  Business  Goals/Strategy 
Organizational  Leadership 
>  Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Making  Performance  Improvement 
Intrinsic  to  the  Job 


STRENGTH  TIIKUEGil  INOLSTRV  &  TECHNOLOGV 


Process  improvement  is  everyone’s 
responsibility 

•  “Quality  is  not  an  act,  it  is  a  habit”  (Doming) 

•  Set  expectations  for  organization-wide  involvement 

-  Managers  at  all  levels 

-  Process  groups 

-  Practitioners  and  support  groups 

•  Establish  mechanisms  for  a  learning  organization 

-  Improvement  suggestions,  lessons  learned, 
process  assets 


Engage  practitioners 

•  The  most  useful  processes  are  often  developed  by 
those  doing  the  work  -  not  “ivory  tower”  process  groups 

•  Ensure  connection  to  the  real  issues  faced  by  projects 

Involve  respected  experts  and  opinion  leaders 

•  Ensure  process  relevance,  ownership,  buy-in 


Economical  Business  Application 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
>  Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 


The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 


10 


Economical  Implementation  of  CMMI 

(Part  2) 


STRENGTH  T1IKUEGU INULSTRV  &  TECHNOLOGV 


Section 

Topics 

Economical 

Implementation  of  CMMI 
(Implementers) 

•  Use  CMMI  as  an  Integrating  Framework 

•  Develop  and  Deploy  Processes  Effectively 

•  Tailor  CMMI  Implementation  Appropriately 

•  Implement  CMMI  in  a  Practical  Way 

•  Make  an  Informed  Decision  on  High  Maturity 

•  Conduct  Appraisals  Economically 

Practical  guidance  for  implementing  CMMI  economically 

•  Helps  ensure  investments  yield  returns  in  business  performance 

•  Recommendations  for  effective  implementations  to  avoid  common  pitfalls 

•  Non-exhaustive,  perhaps  subject  to  debate  -  intended  to  be  interpreted, 
tailored  and  applied  in  business  context 

Intent  is  to  help  maintain  CMMI  emphasis  where  it  belongs 

•  Improvement  in  business  results  and  project  performance, 
achieved  economically 
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Use  CMMI  as  an  Integrating  Framework 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLUGV 


Common  Issues 

Recommendations 

•  Multiple  parallel  improvement 
strategies  (e.g.,  CMMI,  ISO,  Lean,  Six 
Sigma)  not  well  coordinated  at  the 
organizational  level. 

•  Not  all  functions  engage  in  integrated 
process  improvement,  resulting  in 
sub-optimized  processes  or  disjoint 
initiatives. 

•  Create  one  set  of  oroanizational  process 
standards  suoportinq  multiple  improvement 
strategies.  Use  CMMI  to  create  a  process 
architecture  and  framework  supporting  multiple 
process  guidance  sources. 

•  Inteqrate  stakeholders  and  cross-functional 

processes  usinq  CMMI  to  identify  issues  earlv 
in  the  product  life  cycle. 

CMMI  can  be  used  to 
integrate  processes, 
stakeholders  and 
improvement  initiatives 


Economical  Business  Application 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
>  CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Develop  and  Deploy  Processes 
Effectively 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLUGV 


Common  Issues 

Recommendations 

•  Processes  too  closely  aligned  with 
CMMI  model  don’t  fit  the  organization 

•  Processes  developed  in  isolation  from 
projects  aren’t  realistic  or  accepted 

•  Too  much  change  at  once 
overwhelms  the  organization 

•  Process  descriptions  are  too  verbose, 
disorganized,  or  overly  dependent  on 
manual  effort  to  be  useful  to  projects 

•  Inteqrate  CMMI  with  current  practices.  Desiqn 
processes  around  work  actually  performed. 

•  Involve  practitioners  to  help  develop  and 
deploy  processes  that  are  practical  and  useful. 

•  Manaqe  the  improvement  initiatives.  Consider 
improvement  lifecycles.  Pilot  for  effectiveness. 

•  Maintain  perspective  -  remember  who 
processes  are  for,  and  why.  Keep  end  users  in 
mind  as  the  primary  target  for  useful,  concise 
process  descriptions  ready  to  be  followed 

Design  processes  so 
they  are  effective  and 
most  usefui  to  those 
that  must  foiiow  them 


Economical  Business  Application 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
>  Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Tailor  the  CMMI  Implementation 
Appropriately 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLOGV 


Common  Issues 

Recommendations 

•  Organizations  adapting  to  CMMI, 
instead  of  adapting  CMMI  to  their 
business 

•  Forcing  a  “one  size  fits  all”  CMMI 
implementation  on  the  diverse 
projects  in  the  organization 

•  Tailor  CMMI  model  implementation  to  the 
business  context.  Adapt  CMMI  implementations 
to  meet  the  needs  of  the  business. 

•  Recoanize  the  needs  of  different  tvoes  of 
proiects.  Allow  and  encouraae  proiect  tailorina 
of  the  organization’s  process. 

CMMI  is  a  model,  not  a 
process  -  adapt  it  to  fit 
the  characteristics  and 
constraints  of  the 
business  context 


Economical  Business  Application 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
>  Tailor  CMMI  Appropriately 

Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Implement  the  CMMI  in  a  Practical  Way 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLUGV 


Common  Issues 

Recommendations 

•  Size  of  the  CMMI  model  can  be 
overwhelming  for  newcomers. 

•  Confusion  about  generic  practices 
causes  process  rework. 

•  Inability  to  estimate  process 
improvement  effort  causes  cost  and 
schedule  problems. 

•  Start  simolv  and  bite  off  manaoeable  chunks. 

Identify  areas  where  needs  are  greatest. 
Understand  model  dependencies. 

•  Interpret  and  aoolv  CMMI  generic  practices 
with  good  iudgment.  Find  practical  solutions  for 
implementation/appraisal  that  support  the  work. 

•  Learn  from  experience.  Collect  measures  for 
improvement  cost  and  effort.  Use  training  and 
other  resources  to  minimize  misunderstandings 
that  can  cause  rework. 

Use  good  judgment  on 
CMMI  implementation 
strategies  to  manage 
compiexity  and  maximize 
business  ieverage 


Economical  Business  Appiication 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economicai  impiementation  of  CMMi 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
>  Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Make  an  Informed  Decision  on  High 
Maturity 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLOGV 


Common  Issues 

Recommendations 

•  Misunderstanding  high  maturity  leads 
to  folklore  on  burdensome  processes. 

•  Focus  on  high  maturity  level  ratings 
over  actual  improvement  value. 

•  Concern  that  high  maturity  requires 
excessive  rework  of  processes. 

•  Un-measurable  quality  and  process 
performance  objectives. 

•  Settling  for  MLS,  losing  opportunities 
for  greater  business  leverage. 

•  Separate  fact  from  fiction.  Take  traininq  to 
understand  high  maturity  and  find  opportunities. 

•  Focus  on  process  improvement,  not  maturity 

levels. 

•  Anticipate  process  evolution.  Plan  for  natural 
progression  of  improvement,  at  any  level. 

•  Derive  measurable  qualitv  and  process 
performance  obiectives  from  business  needs. 

•  Make  an  informed  decision  on  hiqh  maturity. 

Seek  first  to  understand  ,  then  determine  where 
it  makes  sense  for  the  business. 

Greatest  business  benefit 
can  be  obtained  by 
impiementing  the  appropriate 
ievei  of  process  maturity 
based  on  business 
objectives 


^  5.  Optimizing 

4.  Quantitatively  Managed 
3.  Defined 


Economical  Business  Application 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economical  Implementation  of  CMMI 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
>  Informed  Decisions  on  High  Maturity 
Conduct  Appraisals  Economically 
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Conduct  Appraisals  Economically 


STRENGTH  T1IKULGU INDLSTRV  &  TECHNOLUGV 


Common  Issues 

Recommendations 

•  Behaviors  based  on  fear  of  failing 
ratings  drives  disproportionate  effort 
on  appraisal  preparation  and  dry  runs. 

•  Focusing  on  appraisal  ratings  and  not 
acting  upon  improvements. 

•  Expensive  appraisals,  preparation  and 
evidence  collection  can  burden  CMMI 
adoption. 

•  Appraisals  of  supplier  processes  can 
be  cost-prohibitive  in  acquisition. 

•  Utilize  the  entire  family  of  aooraisal  methods 

(Class  A,  B,  C)  aoDrooriatelv  -  right  tool  for  the 
right  purpose.  Design  an  appraisal  strategy. 

•  Use  appraisals  as  process  improvement 
opportunities  and  as  a  measure  of  progress. 

•  Conduct  efficient  appraisals.  Minimize  creation 
of  evidence  repositories  and  artifacts  intended 
just  for  appraisals. 

•  Use  targeted  appraisals  to  determine  supplier 

processes  risks  most  relevant  to  a  planned 

acguisition.  Look  bevond  ratings  for  suitability. 

Establish  cost-effective 
strategies  for  appraisals  that 
align  with  business  needs 
and  measure  improvement 
progress 


Economical  Business  Appiication 

Support  Business  Goals/Strategy 
Organizational  Leadership 
Improvement  Velocity 
Make  Improvement  Intrinsic  to  the  Job 
Economicai  impiementation  of  CMMi 
CMMI  as  an  Integrating  Framework 
Develop/Deploy  Effective  Processes 
Tailor  CMMI  Appropriately 
Implement  CMMI  in  a  Practical  Way 
Informed  Decisions  on  High  Maturity 
>  Conduct  Appraisals  Economically 


The  Economics  of  CMMI® 
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Summary  - 

The  Economics  of  CMMI 


STRENGTH  TIIKUEGil  INULSTRV  &  TECHNOLUGV 


Business  returns  on  CMMI  investments  are  dependent  iargeiy  on 
underiying  principles 

•  Objectives  -  alignment  with  business  goals 

•  Sponsorship  -  leadership,  commitment,  resources 

•  Action  -  improvement  velocity  for  business  needs 

•  Engagement  -  participation,  project  focused 

•  Value  -  performance  results  to  Justify  investments 

•  Motivation  -  performance  improvement  vs.  ratings 

These  factors  are  under  an  organization’s  control 

•  The  Economics  of  CMMI  is  a  balance  sheet  for  obtaining 
best  value  from  CMMI 

•  Implementation  strategies  govern  whether  CMMI  investments  translate  into 
improved  business  performance,  or  simply  added  costs  of  doing  business 

Focus  on  business  value  to  provoke  thoughtful  dialog  and  raised 
expectations  for  the  effective  use  of  CMMI 


The  Economics  of  CMMI® 
NDIA  CMMI  Working  Group 
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Do  You  Need  CMMI? 


Recognize  these  symptoms? 

•  Missed  commitments 

-  Late  delivery 

-  Last  minute  crunches 

-  Spiraling  costs 

•  Inadequate  management  visibility 

-  Too  many  surprises 

•  Quality  problems 

-  Too  much  rework 

-  Functions  not  working  correctly 

-  Customer  complaints 

•  Poor  morale 

-  Crisis  atmosphere 

-  High  turnover 

-  Low  productivity 


CMMI  for  Executives 
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Does  the  following  occur? 

•  Poor  planning 

-  Plans  not  realistic  or  followed 

-  Work  is  not  tracked  against  the 
plan;  plans  are  not  adjusted. 

•  Baselines  not  controlled 

-  Inconsistent  requirements 

-  Changes  not  managed 

•  Ineffective  organizational  structure 

-  Functions  not  well  integrated 

-  Designs  not  producible 

•  Unable  to  repeat  successes 

-  Staff  skills  and  knowledge  not 
available  when  needed 

-  Dependent  on  heroic  individuals 


CMMI  Features  Help  Address  Common  Issues 


STRENGTH  TIIKUlGil  INDLSTRV  &  TECJLNOLOGV 


CMMI  Feature 

Description  and  Examples 

Results  Oriented 

•  Industry  best  practices  for  project  planning  and  execution 

•  Performance-driven  measures  for  consistent  outcomes 

Priorities  Based  on 
Business  Value 

•  Investments  and  maturity  prioritized  to  align  with  business  goals 

•  Appraisals  relative  to  model  to  set  direction  (“map  and  compass”) 

Customer  Focus 

•  Validation  of  customer  needs  across  the  project  life  cycle 

•  Manage  product/service  quality  (verification,  validation,  reviews) 

Proactive 

Management 

•  Forward-looking  measurement,  monitoring,  risks,  corrective  action 

•  Management  decisions  based  on  plans,  data,  alternatives 

Fiexibiiity 

•  Adaptable  to  a  variety  of  businesses  (domain,  size,  products) 

•  Non-prescriptive  (required,  expected,  informative  components) 

Business  Process 
Integration 

•  Cross-functional  stakeholder  involvement 

•  Coordinate  various  improvement  strategies  and  methods 
(Lean,  Six  Sigma,  ISO,  Agile,  etc.) 

Continuous 

Learning 

•  Standardized  assets  tailored  for  project  characteristics 

•  Leverage  experience  and  history  across  projects 

CMMI  for  Executives 
October  2009 
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Why  Focus  on  Process? 
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PEOPLE 


The  quality  of  a  system  is  highiy  influenced  by  the  quaiity  of  the 
process  used  to  acquire,  deveiop,  and  maintain  it. 

•  A  long-standing  premise  in  manufacturing 

•  Good  processes  increase  the  likelihood  of 
successful  projects 

Process  can  enhance  the  capabiiities 
of  your  workforce 

•  Work  smarter,  not  just  harder 

•  Leverage  organizational  experience 
and  best  practices 

Process  integrates  technoiogy 
with  resources 

•  Technology,  by  itself,  will  most 
likely  not  be  used  effectively 


TECHNOLOGY 
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What  Is  CMMI? 
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CMMI  is  a  model  representing  a  collection  of  best  practices  proven 
effective  in  industry 

•  A  framework  for  developing,  improving,  and  sustaining  business  performance 

•  Provides  a  process  focus  on  work  activities 

•  Developed  by  industry  (commercial  and  defense),  government,  academia 

CMMI  targets  three  primary  environments: 


•  Development  - 

Engineering  a  product  or  service 

•  Services  - 

Providing  services 

•  Acquisition  - 

Acquiring  products  and  services 

The  CMMI  product  suite 
consists  of: 

•  Models  and  primers 

•  Appraisal  methods 


CMMI-DEV 

n 

CMMI  Model 

1 

i 

Framework 

1 

< 

n 

r _ y 


•  Training  courses 


Capability  Maturity  Model  Integration  (CMMI®) 
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What  CMMI  Can  Add  to  Your  Organization 
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•  Integration  of  business  processes  across  functions  based  on  industry 
best  practices 

•  Visible  project  and  organizational  measures  aligned  with 
achievement  of  business  objectives 

•  Commonly  accepted  process  framework  for  inter-company 
coordination  and  competitor  benchmarking 

•  Repeat  project  successes  through  standardization,  tailoring,  and 
capture  of  organizational  process  assets 

•  Avoid  project  performance  issues  through  process  discipline, 
proactive  management,  and  early  stakeholder  engagement 

•  Predictable  project  performance,  with  fewer  surprises 


CMMI  for  Executives 
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CMMI  Model  Overview 
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Process 

Areas 

Clusters  of  related  practices,  in  several  categories 
•Project  Management  -  planning,  monitoring,  suppliers,  risk,  ... 
•Support  -  CM,  QA,  measurement,  decision  analysis,  ... 

•Process  Management  -  organizational  processes,  training,  ... 
•Engineering  -  requirements,  development,  integration,  ... 

•Services  -  development,  delivery,  transition,  ... 

•Acquisition  -  requirements,  solicitation,  agreements,  ... 

Generic 

Practices 

Enable  process  management,  deployment  and  improvement 

•Plans,  monitoring,  CM,  stakeholders,  objective  evaluation,  ... 

Goais 

Describes  characteristics  for  implemented  processes 

Capabiiity 

Leveis 

Achievement  of  process  improvement  within  an  individual 
process  area 

Maturity 

Leveis 

Achievement  of  process  improvement  across  a  predefined  set 
of  process  areas  (stages) 

CMMI  for  Executives 
October  2009 
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CMMI  Appraisals 
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Appraisals  compare  organization  and  project  processes 
against  CMMI  models  to  determine  improvement  priorities 

Senior  management’s  role  in  appraisals: 

•  Provide  sponsorship  and  resources 

•  Set  appraisal  scope  and  objectives 

•  Ensure  follow-through  on  appraisal  findings  and  prioritized 
improvement  actions 

CMMI  provides  a  family  of  appraisal  methods,  with  varying 
intent,  confidence  levels,  data  collection,  resources  needed 

•  Flexible  focus:  approach,  deployment,  institutionalization 

•  Rigorous  benchmark  rating  method  (for  maturity  levels) 

•  “Quick  look”  diagnosis  of  process  weaknesses 

Licensed  SEI  partners  deliver  SCAMPI^i^  appraisal  services 

•  http://www.sei.cmu.edU/collaboratinq/partners/cmmiv1.2/ 
Note  that  for  internal  process  improvement,  company- 
developed  and  other  methods  can  be  effective 


standard  CMMl" 
Apprat&al  Method  for 
Process  Improvement 
(SCAMPI^“)A, 
Version  1.2: 

Method  Definition 
Document 


SEI  Partner 

CMMI 
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Reasons  You  Should  Adopt  CMMI 
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1.  Increase  customer  satisfaction 

•  Deliver  products  and  services  that  satisfy  user  needs 

•  Deliver  products  and  services  on  time  and  within  budget 

2.  Increase  probability  of  capturing  new  and  repeat  business 

•  Improved  ability  to  meet  commitments 

•  Reduces  customer-perceived  risk  of  award  to  your  organization 

•  Can  be  a  discriminator  relative  to  your  competition 

3.  Increase  profit  through  improved  quality  and  less  rework 

•  Better  predict  actual  costs  through  repeatable  processes 

•  Better  visibility  into  projects  due  to  established  measures  and  analysis  techniques 

•  Significantly  reduce  the  probability  of  problem  programs 

•  Reduce  costs  by  capitalizing  on  organizational  infrastructure,  processes,  training, 
tools  and  early/often  stakeholder  involvement 

4.  Increase  productivity 

•  More  efficiency  through  implementation  of  common  processes,  tools  and  training 

•  Improved  productivity  by  implementing  process  improvement  that  are  directly 
aligned  key  organizational  goals  and  objectives. 

•  Higher  employee  morale  and  less  turnover 
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Benefits  of  CMMI-Based  Process  Improvement 
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Many  companies  cite  performance 
benefits  from  CMMi 


•  Published  in  conferences,  articles, 
papers,  studies,  surveys,  reports 

SEi  coiiects  quantitative  measures 
of  CMMI  performance  improvement 

•  Technical  reports,  including: 

-  “Performance  Results  of  CMMI- 
Based  Process  Improvement” 
(http://www.sei.cmu.edu/pub/docume 

nts/06.reports/pdf/06tr004.pdf) 


Performance 

Median 

Category 

Improvement 

Cost 

34% 

Schedule 

50% 

Productivity 

61% 

Quality 

48% 

Customer 

Satisfaction 

14% 

ROI 

4.0  :  1 

CMU/SEI-2006-TR-004. 

Data  from  35  organizations. 
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CMMI  Adoption 
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CMMI 
appraisals 
are  conducted 
worldwide... 


USA  Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

354 

33.6% 

2566 

91.3% 

586 

55.7% 

183 

6.5% 

113 

10.7% 

63 

2.2% 

1053 

'  100.0% 

2812 

100.0% 

...in  a  wide  range  of  businesses 


...in  small 
and  large 
organizations 
and  projects 


>200, 

25.2% 

101-200, 

18.9% 


<100, 

53.5% 


>2000, 1.9% 
1001-2000,3.3% 
501-1000,  5.6% 
301-500,  6.7% 
201-300,  8.0% 

101-200,  18.9% 
76-100,  8.3% 
51-75,  12.8% 

26-50,  19.9% 
<25, 15.0% 


Organization  Size  (Empioyees) 

(3863  organizations  reporting) 


...at  all  levels  of  process  maturity 


Services  (70.1%) 

•  Business  Services 

•  Engineering  and 
Management  Services 

•  Health  Services 

•  Other  Services 


Manufacturing  (16.8%) 

•  Electronic  and  Electric  Equipt 

•  Transportation  Equipment 

•  Instruments  &  Related  Products 

•  Industrial  Machinery 

•  Other  Mfg  Industries 


Other  (13.1%) 

•  Finance,  Insurance,  Real  Estate 

•  Public  Administration/Defense 

•  Transportation,  Communication, 
Utilities 

Based  on  primary  Standard  Industrial  Classification  (SIC)  codes  reported  in 
CMMI-based  appraisals. 


Co  n  tra  cto  r  M  i  I  i  ta  ry/ 

Commercial  for  Military/  Government 
In-House  Government  Agency 


No  Rating  Given 

5.7% 

8.5% 

22.7% 

Initial  (ML1) 

0.8% 

1.6% 

1.7% 

Managed  (ML2) 

28.0% 

31.7% 

44.3% 

Defined  (ML3) 

53.6% 

46.7% 

26.7% 

Quantitatively  Managed  (ML4) 

3.1% 

1.4% 

1.1% 

Optimizing  (ML5) 

8.7% 

10.1% 

3.4% 

(2920  orgs) 

(769  orgs) 

(176  orgs) 

Source:  SEI  Process  Maturity  Profile,  Sept  2009. 
http://www.sei.cmu.edu/appraisal-proqram/profile/ 
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Getting  Value  from  CMMI 

Your  Role  as  an  Executive 
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Set  the  vision  and  direction  for  CMMI-based  improvement 

•  Establish  measurable  objectives 

•  Be  a  visible  sponsor  -  set  expectations  for  involvement 

•  Manage  process  improvement  like  a  project 

Provide  resources  and  support 

•  Funding,  staffing,  tools 

•  Choose  the  best  people  to  lead  -  respected  opinion  leaders 

Keep  it  reai 

•  Maintain  relentless  focus  on  business  value  and  program  performance 

•  Involve  projects  and  practitioners  for  the  best  ideas 

•  Hold  people  accountable 

•  Track  and  communicate  progress 

•  Recognize  and  reward  achievement 
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The  Effective  Use  of  CMMI® 
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Summary  of  NDIA  industry  position  statements  for  obtaining  best  vaiue 
from  CMMI  investments*: 


1.  Good  processes  increase  the  likelihood  of  achieving  successful  project  performance 

2.  CMMI  is  a  model,  not  a  standard  -  adapt  CMMI  to  your  business  environment, 
resources,  and  objectives 

3.  Focus  on  business  improvement  objectives  -  a  primary  emphasis  on  achieving 
levels  may  not  achieve  significant  benefits  and  may  increase  rather  than  decrease 
costs 

4.  High  maturity  is  a  business  case  -  justify  the  investment:  many  organizations  find 
business  value  in  improving  processes  even  at  lower  CMMi  maturity  levels 

5.  Maturity  level  ratings  are  not  alone  a  predictor  of  project  performance  -  many 
other  factors  can  be  significant  contributors 

6.  Don’t  specify  maturity  levels  in  acquisitions  -  use  CMMI  to  probe  supplier 
capability  and  process  execution  risks 

7.  Greatest  benefits  of  appraisals  are  from  improvements,  not  evidence  or  ratings  - 

disproportionate  effort  on  appraisal  preparation  risk  can  diminish  business  returns 


“The  Effective  Use  of  CMMI®”,  NDIA  Systems  Engineering  Division,  June  2009. 
httD://wvvw.ndia.ora/Divisions/Divisions/SvstemsEnaineerina/Paaes/CMMI  Working  Group.aspx 
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Want  to  Learn  More  about  CMMI? 


STRENGTH  TIIKUlGil  INULSTRV  &  TECJLNOLOGV 


SEI  CMMI  web  pages: 


What  is  CMMI?  Models 

Conferences  Performance  Results 

FAQs  Background  Information 


CMMI  focus  topics,  guidance,  technicai  reports: 


CMMI  and  Agile 
CMMI  in  Small  Settings 
Earned  Value  Management 

Training: 


CMMI  and  Six  Sigma 
CMMI  in  Acquisition 
SW-Only  Organizations 


Process  Improvement  Introduction  to  CMMI 

CMMI  Level  2-3  for  Practitioners  Understanding  High  Maturity 

User  Networks 


SEI  Partner  Network 
Consultants 


Newsgroups,  Blogs,  Wikis 
Conferences 


Questions?  Comments? 


Adoption 

Appraisals 

Contacts 


Product  Line  Practices 
Interpretive  Guidance 
Operations  Organizations 


Intermediate  Concepts  of  CMMI 
SCAMPI  Appraiser  training 


Books,  Periodicals,  Articles 
Asset  Repositories 


Web:  http://www.sei.cmu.edu/cmmi 
Email:  cmmi-comments@sei.cmu.edu 

SEI  Customer  Relations:  (412)  268-5800,  customer-relations@sei.cmu.edu 
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Assurance  for  CMMI®:  A  Toolbox  for  Multiple 

Cyber  Challenges 


9th  Annual  CMMI®  Technology  Conference 
17  November  2009 

Michele  Moss,  Booz  Allen  Hamilton 
Debbie  McCoy,  Booz  Allen  Hamilton 


®CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


•  Setting  the  Stage 


•  Assurance  for  CMMI® 

•  Code  Vulnerabilities 

•  Global  Supply  Chain 

•  Organizational  Cyberspace 

•  Next  Steps 
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Today’s  Reality  Requires  Increased  Confidence  In  Our  IT 

Products  and  Services  .  . . 


•  Dependencies  on  technology 
are  greater  then  ever 

•  Possibility  of  disruption  is 
greater  than  ever  because 
software  is  vulnerable 

•  Loss  of  confidence  alone  can 
lead  to  stakeholder  actions 
that  disrupt  critical  business 
activities 


Control  Systems 
>  SCADA 
>PCS 
>XS 


>  Agriculture  and  Food 

>  Energy 

>  Transportation 

>  Chemical  Industry 

>  Postal  and  Shipping 


>  Water 

>  Public  Health 

>  Telecommunications 

>  Banking  and  Finance 

>  Key  Assets 


Critical  Infrastructure  /  Key  Resources 


Physical  Infrastructure 

Services  Hardware 

>  Managed  Secunty  >  Database  Servers 

>  Information  Services  >  Networking  Equipment 


>  ReservoirsTreatment  Plants 

>  Farms 

>  Food  Processing  Plants 

>  Hospitals 

>  Power  Plants 

>  Production  Sites 


>  Railroad  Tracks 

>  Highway  Bridges 

>  Pipelines 

>  Ports 

>  Cable 

>  Fiber 


>  FDIC  institutions 

>  Chemical  Plants 

>  Delivery  Sites 

>  Nuclear  Power  Plants 

>  Government  facilities 
>Dams 


Internet 

>  Domain  Name  System 

>  Web  Hosting 


Cyber  Infrastructure 
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Adapted  from:  Jarzombek,  Mitigating  Risks  to  the  Enterprise  through  Development  and  Acquisition,  SEPG  2009 


Gaps  Exist  In  The  Intended  Audience  For  Sw/A  Literature-  'Ji 


Distribution  of  Literature 

(Percen^§gAa£J^lerature  by  Intended  Audience) 


|tandards  Community 
0.4% 


Academia 
21 .3% 


Academia  and 
Practioners 
17.0% 


Academia  and 
Practitioners 
1 .5% 


June  2009  SwA  Working  Group  Session  -  Courtesy  of  Jeff  Inglesbe 


•  Setting  the  Stage 

•  Assurance  for  CMMI® 


•  Code  Vulnerabilities 

•  Global  Supply  Chain 

•  Organizational  Cyberspace 

•  Next  Steps 
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Policy 


/  Processes 
/  for  Assurance  \ 

Project  leadership  and  team 
members  need  to  know  where 
and  how  to  contribute 

/  Methodologies  \ 

/  For  achieving  Assurance  \ 

Focus  Topic:  Assurance  for 
CMMI®  defines  the  Assurance 
Thread  for  Implementation  and 
Improvement  of  Assurance 
Practices 

/  Detailed  Criteria  \ 

https://buildsecuritvin.us-cert.qov/swa/procresrc.html 
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Assurance  Focus  -  Organizational  Training 


Tha  puipD  M  of  Qrsnizatinnal  TfaiD-io  s  (OT)  i£  to  dav  alDp  hs  EkilE  and  tnmvlad  Ea  of  paopla  ed 
th^'  can  parionn  tteirmlas  affactiv’a^'  and afflc-Eniy.  [L  p.  275] 


otganiiatioyi 's framiyig  n^sdi:  Ae  liy^ith^od  £^aiqijai\fied 

and  i^ptupral^iy  If  a  in^d  r^::ouK^::  ar^p^atm  ing  Jfiff  nsc^::::aty  mi^gtai^d  az::iirancf^ 
actbAti^^  on  ihepjoJ^cL 


Context  of  Assurance  for  the  PA 


Th^  us^of  itv  Foca::  'F?pic  a^d€::cribsd  ihjvughoiit  d:foiim!a^t  cKai^::a  yk2lura\  inch::ton 
of  as^mance actix-ttios  for  finontng pKJciicos  ■H.tihin  6^0  OT  ptooe^^aroa:  SF1.2,  SP1.4, 


AEEUianca  j^Easana^t  i^nak-EiE 
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Assurance  Risks  and  Software  Quality 


•  64%  of  the  vulnerabilities  in  NVD  in  2004  are  due  to 
programming  errors* 

-  51  %  of  those  due  to  classic  errors  like  buffer  overflows, 
cross-site-scripting,  injection  flaws* 

•  Probability  of  serious  vulnerabilities  is  52.3%  (Capers 
Jones  Overview  of  the  US  software  Industry,  April 
2008) 

•  27%  of  development  effort  is  devoted  to  defect 
removal,  repair,  and  rework  (Capers  Jones  Overview  of 
the  US  software  Industry,  April  2008) 

•  67%  percent  of  the  attacks  in  2007  were  "for  profit" 
motivated,  ideological  hacking  came  second  (Web 
Application  Security  Consortium  Annual  2007  Report) 


Risk 


Reputation 

Litigation 

Compliance 

Service 

Productivity 

Technology 


*  courtesy  of  Robert  Seacord 
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r 


Technology: 

Static  Analysis, 
CWE,  CVE,  CVSS 


(  \ 

People: 

Project  manager 
Security  analyst 
Developer 

V _ y 


V _ J 


Assurance  for 
CMMI®  Practice 

TSAF  3.1.2 
Identify  deviations 
from  assurance 
coding  standards 
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SDLC 

Activity 

Assurance  for  CMMI 

BSIMM 

TSP  Secure  * 

Code  Review 
Checklists 

OPD  AF  1.1.1  Establish  and 
maintain  organizational 
processes  to  achieve  the 
assurance  business 
objectives. 

TSAF  3.1.2  Identify 
deviations  from  assurance 
coding  standards. 

SR  Level  1 :  Provide  easily 
accessible  security  standards 
and  (compliance-driven) 
requirements 

CERT  SCI  provides  language 
specific  secure  coding  guidelines  for 

C,  C++,  and  Java. 

To  claim  compliance  with  a  standard, 
software  developers  must  be  able  to 
produce  on  request  documentation 
as  to  which  systematic  and  specific 
deviations  have  been  permitted 
during  development. 

Static  Analysis 
Tools 

iPMAF  1.3.1  Establish  and 
maintain  assurance  of  the 
project’s  work  environment 
based  on  the  organization’s 
work  environment  standards. 

CR  Level  2:  Enforce 
standards  through  mandatory 
automated  code  review  and 
centralized  reporting 

CR  Level  3:  Build  an 
automated  code  review 
factory  with  tailored  rules 

Automatable  guidelines  are  identified 
by  WG14/N1393.  Remaining 
guidelines  are  enforced  through 
manual  inspection.  The  CERT 

Source  Code  Analysis  Laboratory 
certifies  conformance  to  standards. 

courtesy  of  Robert  Seacord 


•  Setting  the  Stage 

•  Assurance  for  CMMI® 

•  Code  Vulnerabilities _ 

•  Global  Supply  Chain 
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•  Deliberately  embedded  malicious  functionality 

•  Theft  to  intellectual  property 

•  Fake  or  counterfeit  products 

•  Exploitable  IT/software  unintentionally  produced  by 
suppliers  with  poor  security  practices 

•  Lack  of  developer  and  acquirer  awareness  of  associated 
risks 


Increased  Vigilance  Is  Critical  To  Reducing  IT 
Risks  From  The  Suppiy  Chain 


Adopted  from  Marcus  H.  Sachs,  Verizon,  ’’Supply  Chain  Risk  Management:  Can  we  Secure 
the  IT  Supply  Chain  in  the  Age  of  Globalization?”  Software  Assurance  Forum,  15  Oct  2008 
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f  ^ 

Technology: 

Automated  Penetration 
Testing  Tools 

Network  Vulnerability 
Scanners 

V _ J 


People: 

Project  manager 
Security  analyst 
Developer 

V _ ) 


Assurance  for  CMMI®  Practice: 


TS  AF  2. 1. 1  Architect  for  assurance. 

TSAF  2.1.2  Design  for  assurance. 

TS  AF  3.1.1  Implement  the  assurance 
designs  of  the  product  components. 

VAL  AF  2.2. 1  Analyze  the  results  of 
assurance  validation  activities. 


VER  AF  3.2. 1  Analyze  the  results  of 
assurance  verification  activities. 
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Software  Supply  Chain  Integrity 


*  Established  Design  Principles 

-  Chain  of  Custody:  The  confidence  that  each  change  and  handoff  made  during  the  source 
code’s  lifetime  is  authorized,  transparent  and  verifiable. 

-  Least  Privilege  Access:  Personnel  can  access  critical  data  with  only  the  privileges  needed 
to  do  their  jobs. 

-  Separation  of  Duties:  Personnel  cannot  unilaterally  change  data,  nor  unilaterally  control 
the  development  process. 

-  Tamper  Resistance  and  Evidence:  Attempts  to  tamper  are  obstructed,  and  when  they 
occur  they  are  evident  and  reversible. 

-  Persistent  Protection:  Critical  data  is  protected  in  ways  that  remain  effective  even  if 
removed  from  the  development  location. 

-  Compliance  Management:  The  success  of  the  protections  can  be  continually  and 
independently  confirmed. 

-  Code  Testing  and  Verification:  Methods  for  code  inspection  are  applied  and  suspicious 
code  is  detected. 


The  Software  Supply  Chain  Integrity  Framework  Defining  Risks  and  Responsibilities  for  Securing  Software  in  the 
Global  Supply  Chain  http://www.safecode.org/publications/SAFECode_Supply_Chain0709.pdf 


15 


•  Setting  the  Stage 

•  Assurance  for  CMMI® 

•  Code  Vulnerabilities 

•  Global  Supply  Chain 

•  Organizational  Cyberspace 

•  Next  Steps 


16 


Stovepiped  Assurance  Efforts  Miss  The  Dartboard  ^ 


1 

j 


Component  'y 
||  Integration  &  11 

—i  - - 1 

ble  n 

Lre  J 


■  Code  Software 


Concept 


i  INTERATION 

*  Planninr^  INTEj^TION 

Plannin  ^EASE 

DAILY 
'fesj^n 

Close  Oijt  y 
Close  Oijt 


Deployrqejpj^ 


1  Close  Outj  y/ 


Deployment 


p  ^ 


■r  \ 
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/  \ 

Technology: 

Process, 

Measurement,  and 
Artifact  Repositories 

Social  Media 

V _ y 


People: 

Executive  Sponsors 
Project  Managers 
Project  Teams 

V _ y 


Assurance  for  CMMI®  Practice 

OPFAF  1. 1. 1  Establish  and 
maintain  the  description  of  the 
assurance  context  and 
objectives  for  the  organization. 

OPD  AF  1. 1. 1  Establish  and 
maintain  organizational 
processes  to  achieve  the 
assurance  business  objectives. 

OTAF  1.1.1  Establish  and 
maintain  the  strategic 
assurance  training  needs  of  the 
organization 


V _ y 
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BSIMSR  Level  1:  Provide  easily  accessible  security 
standards  and  (compliance-driven)  requirements 
Safecode  Whitepaper  -  Fundamental  Practices  for 
Secure  SW  Development  (section  on  Programming) 


Methodologies 
For  Achieving  Assurance 


Assurance  for  CMMI®  Provides  the  Framework  to  Connect 
Development  Activities  to  Assurance  Goals  , 


Establish  and  maintain  organizational  processes  to 
achieve  the  assurance  business  objectives. 
Identify  deviations  from  assurance  coding  standards. 
(Source:  Assurance  for  CMMP  March  2009) 


“It  Is  the  policy  of  Motorola  to  offer  security  solutions  designed  to 
protect  the  confidentiality,  integrity  and  availability  of  information  and 
other  assets  appropriate  to  their  value  to  Motorola,  and  to  service 
providers  (and  their  customers)  using  Motorola  products.”  (source: 
Motorola  Secure  Software  Development  Model  (MSSDM)  Lessons 
Learned,  Margaret  Nadworny,  August  10,  2007) 


TSP  Secure  CERT  SCI  provides  language  specific 
secure  coding  guidelines  for  C,  C++,  and  Java. 
To  claim  compliance  with  a  standard,  software 
developers  must  be  able  to  produce  on  request 
documentation  as  to  which  systematic  and  specific 
deviations  have  been  permitted  during 
development. 


Detailed  Criteria 


i>si 

SCAMPI^'^  Is  A  Tool  To  Identify  Assurance  Process  \\ 

Institutionalization  Risks  . 


Plan  and  Prepare  for  Appraisal 


Analyze 

Requirements 


Develo 


^Appraisal 


Obtain  and  Analyze 
Initial  Objective 
Evidence 


Conduct  Appraisal 


Examine  Objective 
Evidence 


Verify  and  Validate 
Objective  Evidence 


Incorporate  Assurance  Focus  Practices 


Report  Results 


Results 


raisal 


Select  and  Prepare 
Appraisal  Team 


Prepare  for 
Collection  of 
Objective  Evidence 


Document  Objective 
Evidence 


Generate  Appraisal 
Results 


Package  and 
Archive  Appraisal 
Assets 


SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University 
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•  Use  “Draft  Practices”  to  identify  gaps  in  your  assurance 
practices  httDs://buildsecuritvin.  us¬ 
ee  rt.  go  v/swa/orocresre.  htmi 

•  Measure  and  improve  your  assurance  practices 

•  Share  your  lessons  learned  (swawa-orocess  @.  cert. ora) 
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References  for  Integrating  Assurance 


DHS  Software  Assurance  Working 
Groups 

-  https://buildsecuritvin.us-cert.qov 

-  http://www.us-cert.qov/swa/ 

lATAC  /DACS 

-  http://iac.dtic/iatac 

-  https://www.thedacs.com 

-  Enhancing  the  Development  Life 
Cycle  to  Produce  Secure  Software 

-  State  of  the  Art  Report  on 
Software  Security  Assurance 

NIST 

-  http://csrc.nist.qov/ 


•  NDIA 

-  Systems  Enqineerinq  Division 

-  System  Assurance  Guidebook 

•  SANS 

-  http://www.sans.orq/ 

•  International  Organization  for 
Standardization  (ISO) 

-  http://www.iso.orq 

•  Software  Security  Engineering 

-  http://www.softwaresecuritvenqine 

erinq.com/ 

-  http://www.amazon.com/Software- 

Securitv-Enqineerinq-Proiect- 
Manaqers/d  p/0321 5091 7X  03 


r  -  -M- 


Contact  Information 


•  Michele  Moss,  CISSP,  CSSLP 
Booz  Allen  Hamilton 

Co-Chair  DHS  SwA  Processes  and  Practices  Working  Group 
moss  michele@bah.com 


•  Debbie  McCoy,  SCAMPI^'^  B/C  Team  Lead,  Introduction  to  CMMI®  Instructor 
Booz  Allen  Hamilton 
mccov  debbie@bah.com 
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f>roc:ess 

CO  IM  S  LJ  LTA ISITS  LL-C 


9*^  Annual  National  Defense  Industrial  Association 
CMMI  Technology  Conference  and  User  Group 

November  17,  2009 
Denver,  Colorado,  USA 

CREATIVELY  APPLYING 

CMMI-SVC 

IN  A  VERY  SMALL 
CONSULTING  FIRM 

Bill  Smith,  CEO 
Leading  Edge  Process  Consultants  LLC 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


Objective  of  This  Presentation 


To  provide  a  glimpse  into  how  one  very 
small  company  is,  little-by-little, 
adopting  key  principles  from  the  CMMI 
for  Services  (CMMI-SVC)  to 
dramatically  improve  its  bottom  line. 


Copyright  2009,  Leading  Edge  Process  Consultants  CmmiTraining.com 
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Agenda 

®  Background 
®  Making  the  Decision 
®  Improving  Our  Marketing 
®  Improving  Our  Training  Deiivery 
®  The  Future 
®  In  Conclusion 


j=  -> 

PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  i-L-C 
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PROCESS 

CO  ISI S  LJ  l_T/\  IM  TS  LL-C 


Who  Are  We,  and  Why  Do  You  Care?  (Or  Not.) 

Background 


Who  Are  We? 


PROCESS 
CIOIMSLJLT/VIMTS  - 


®  Leading  Edge  Process  Consultants^  a  sz/gw 
is  a  well-established,  world-class  exaggeration 

provider  of  Arocess  innproverhe«l 

"x  Depends  on 

consulting,  appraisal,  and  training  the  day,  realty 

services.  \ 


Eventually...  Dec.  2007!  Someday...  BINGOI 

Award-Winning  CMMI  Training 


Copyright  2009,  Leading  Edge  Process  Consultants  CmmiTraining.com 
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PROCESS 

CIO  IM  S  LJ  LT/V IM  TS 


About  “Public”  Training... 

To  appreciate  this  presentation,  you  need  to  understand  why  we  put 
so  much  time  and  energy  into  pubiic  training 

(other  than  the  fact  that  I  personally  love  it) 


If  you  get  it  right. . .  If  you  do  get  it  right. . . 


. .  .you  could  lose  ■  ■  -you  don’t  have  to  eat  this 

your  shirt!  every  night 


Net  income  from  1  public  training  class  can  be  equivalent  to 
3  to  4  private  classes.  Alternatively, 


Copyright  2009,  Leading  Edge  Process  Consultants  CmmiTraining.com 


Growth  of  Our  CMMI  Training  Business 

“Cinderella  story...  outta  nowhere...” [Caddyshack,  1980] 


> 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  i-L-C 


2007 

2008 

2009 

Google  rank  - 
“cmmi  training” 

NA 

Fell  asleep 
before  finding 

#3 

(after  2  SEI  pages) 

#  Intro  to  CMMI 
students  taught 

0 

89 

223  (projected) 

(191  thru  11/6) 

Average  public 
class  attendance 

NA 

11.3 

25.4*  (thru  11/6) 

Net  income 

Negative 

X 

1 5x  (projected) 
(lOxthru  11/6) 

%  of  income 
from  CMMI 
classes 

NA 

100% 

(CMMI-DEV) 

1 00% 

(CMMI-DEV,  -SVC) 

Includes  three  space-constrained  “sell-outs” 
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But...  How  Are  We  Doing  It? 


A  BIT 

“Just  try 
harder”? 


gOWE 

Natural  business 
growth? 


Insanity:  doing  the  same 
thing  over  and  over 
again  and  expecting 
different  results. 

-A.  Einstein 


In  this 
economy? 


YES! 

Get  better? 


CMMI  . 
for  Services 

■|M  Guidelines 

^Bfor 

Superior 


Eileen  C.  Forrester 
Brandon  L.  Buteau 
Sandy  Shrotn 


Using  the  CMMI 
for  Services  as. 
a  guide. 
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PROCESS 
CO  IM  S  LJ  LT/V IM  TS 


2009:  The  Year  in  Review 


STSM 

SST 

CAM 

Made 

CMMI-SVC  CmmiTraining.com  QPM-ish 
decision  goes  iive 


Straining  my 
availability 


/  X 

m 

m 

Marketing 

- ^ - 

Developed 

marketing 

system 


First  public 
class  in  over  5 
months 


Training 

\ _ y  I 

Moved  to  larger 
public  training 
space 


pp 

SSD 

IRP 

SCON 

SST 

SD 

CMMI-SVC  Process  Areas  appearing  on  this  siide  are  representative 
samples;  elements  of  other  PAs  have  also  been  addressed 

Copyright  2009,  Leading  Edge  Process  Consultants  CmmiTraining.com 


Private  Intro  to  CMMI 
Public  Intro  to  CMMI 

Public  SVC  Supplement 
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“CMMI  in  a  One-Person  Company?  Are  You  Crazy?*” 

Making  the  Decision 


*Those  who  know  me  already  know  the  answer. 


Making  the  Business  Decision  dP" 


®  Shouldn’t  we  practice 
what  we  preach  -  the 
CMMI? 

®  But. . .  we  need  to 
focus  on  making 
money  right  now 

®  Time/resources 
available  to  dedicate 
to  long-term 
improvement: 
zip,  ziich,  nada,  none 


Our  process  improvement  budget 
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Making  the  Business  Decision  (2j 


PROCESS 
CIOIMSLJLT/VIMTS  - 


CMMI  -SVC.  VeRI 


IBION  1  .2  J 


CMMI 
for  Services 


Guidelines 

for 

Superior 

Service 


Eileen  C.  Forrester 
Bnindon  L.  Buteau 
Sandy  Shrum 


“Tell  me  where  it  hurts. . . 


Copyright  2009,  Leading  Edge  Process  Consultants  CmmiTraining.com 


®  No  money,  no  problem! 

1 .  Pick  an  organizational 

point 

2.  Address  the  ,  using 
guidance  from  CMMI- 
SVe  as  appropriate 

3.  Repeat  steps  1  and  2  as 
needed 

®  Focus  on  near-term 
tasks  for  just-in-time 
process  improvement 

®  If  it  ever  looks  like 
CMMI-SVC  =  wrong 
business  decision,  then 
simply 
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Making  the  Business  Decision  (sj 

CMMI  for  Services  Diary 


PROCESS 
CIOIMSLJLT/VIMTS  - 


Services  to  beeom 

„d  ■"»«  '"“s,',"  r,  or  »•'- 

r;re';aU.. 

"S'teioiptov""'"-''"' 
it  for  leg'*""  « 

for  a  ‘level  rating  • 

16  20091 


Desired  location  Actual  location 
of  competition  of  competition 
(Dec  2009)  (Jan  2009) 
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CIO  IM  S  LJ  LT/V IM  TS 


Pinpointing  Our  Pain  (d 

Training? 


BTW,  a  trip  through  the  CMMI  glossary  (service,  product,  end  user,  etc.) 
confirms  that  marketing  may  indeed  be  considered  a  “service.  ” 


High  levels  of  student 
satisfaction 


Inadequate  number  of 
students  to  cover  costs  of 
public  classes 
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Pinpointing  Our 

®  Key  marketing  issues: 

•  Ad  hoc,  reactive 

•  on  Google  Ads 
in  2008: 

money  pit? 

•  Web  site  not 
sufficiently... 
compelling 

•  Personally,  still  a 
relative  “unknown” 

•  /  could  go  on... 
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Not  Hard  to  Do,  Because  Last  Year  It  Stunk 

Improving  Our  Marketing 


Creating  a  Marketing  Service 
Systenn 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS 


Researched  marketing  practices 
Identified  components  and 
subcomponents  of  my  target 
service  system 
Identified  current  and  desired 
states  of  each 

Estimated  development  effort 
Drafted  implementation  schedule 
Began  developing  components 


project 
planning  (PP) 


Level 


Websites 
Blogs 

Social  Networkj 
-Mail  Marketii 
Releases 
“leasts 
^ications 

P®aking/Ne^  . 


Press 


^erti 


Sing 
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Developing  CmmiTraining.com  (d™ 

(Marketing  Service  System  Component  2 . 2) 

The  goal:  “Increase  my  conversion  rate 
(ratio  of  website  visitors  to  registered  students)” 
[from  Student  Attraction  Strategy  2009,  1/5/2009] 


Key  features: 

•  Course  catalog  and  schedule* 

•  Online  student  registration** 

•  Secure  credit  card 
transactions** 

•  Differentiators  (why  us?)* 

•  Search  engine  optimized* 

•  Google  ad-optimized* 

•  new  or  improved  versus  prior  website 

**  by Ampiify  Software,  www.ampiifyiic.com 
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Developing  CmmiTraining.com  (2)“°" 


Old,  money¬ 
sucking  website 


Spiffy  new 
money-making 
website 


Classroom  of 
eager  CMMI 
students 


2/7/09  3/9/09  4/23-4/25/09 


30-day  web  development  “sprint”  6  weeks  promo  time 
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Developing  CmmiTraining.com  op— ^ 

Critical  Success  Factors 

®  stop  “working”  so  much! 

•  Suspended  class 
deliveries  to  focus  on 
this 

®  Agile  development 
methodology 

®  Accurate  effort  estimate, 
based  on 

•  Size  (#  web  pages) 

•  Complexity 
(of  each  page) 

•  Reuse  (existing  website) 

®  A  Validation  Team! 

.  3  former  students,  T-^ey  rocked! 

2  business  associates  ■■ 


SERVICE  SYSTEM 
DEVELOPMENT 

SP  1.1  Develop 

Stakeholder 

Requirements 

SP  2.2  Develop  the 
Design 

SP  3.4  Validate  the 
Service  System 


PROJECT 

PLANNING 

SP  1 .2  Establish 
Estimates  of  Work 
Product  and  Task 
Attributes 

SP  1.3  Define  Project 
Lifecycle 

SP  3.2  Reconcile 
Work  and  Resource 
Levels 
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Measuring  Our  Marketing  Results 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


measurement 

and  analysis 
(MA) 


Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec 


2009  vs.  2008 

Google  Ad  Dollars  -26% 

Click-thru  Rate  +260% 

Avg  Time  on  Website  +44% 

Public  Class  Size  +61% 


Given  our  corporate  vision,  this  may 
have  been  the  difference  between 
staying  in  business...  and  not. 


Comparison  of  4  weeks  prior  to  10/27-10/29/09  and  11/11-11/13/08  ciasses.  Unabie  to  compare  aii  of 
2008  vs.  aii  of  2009  because  not  ail  of  this  data  was  captured  in  2008. 

Private  Intro  to  CMMI,  2009  Mktg 

Sorry,  percentages  only!  The  actual  data  is  proprietary.  Public  Intro  to  CMMI 
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other  Private  Intro  to  CMMI  21 


Moving  Toward  Quantitative  Management°~‘" 


Random  variation,  or 
“speciai  cause”? 

if  speciai  cause, 
eiiminating  it  couid  be 
worth  tens  of  thousands  of 
doiiars 

i  have  a  hunch,  but  iack 
the  correct  data  to  verify 

Coiiecting  that  data  now, 
but  may  not  know  for 
another  year! 


quantitative 

project 

management  (ish) 
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We  Now  Have  Flying  Monkeys...  and  More! 


j=  -> 

PROCESS 
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Improving  Our  Training 
Delivery 


IrtTodocwini 


^wlport"*'’ 


I  create  a 

^**”^^*^  Lfh^speWf^R®*^ 


rtttO(HKi«l 

orreri'O'^ 


Documented  Procedures  Help  Us  to  Grow 


PROCESS 
CIOIMSLJLT/VIMTS  - 


(1) 


Why  “just-in-time”? 

/  rarely  say  to  myself  “I’m  looking  for 
something  to  do  right  now,  so  I  think  I’ll  write 
a  procedure.”  Just-ln-time  works  forme. 


®  August,  2009.  A 

frighteningly  busy  Sept/Oct 
was  looming: 

•  5  Intro  to  CMMI  classes  in  a 
7-week  period 

•  2  of  these  public,  requiring 
tons  of  work;  the  other  3  out- 
of-town 

®  How  to  stay  organized? 

•  Created  procedures  and 
checklists  Just-in-time, 
because  I  really  needed  them 

•  Better  solution  than  constantly 
re-creating  to-do  lists! 
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Documented  Procedures  Help  Us  to  Grow  dp"”— 


®  Thanks  to  procedures, 

checklists,  &  mentoring, 

somebody  else  now: 

•  Creates/prints  certificates 

•  Creates/prints  name  tents 

•  Enters  data  from  SEI 
evaluations  into 
spreadsheets 

•  Assembles  handout  packets 

•  Packs  student  bags  (CM  Ml, 
student  notebook,  freebies) 

•  Restocks  inventory 

®  He’S  12 


Connor  Smith 

Manager,  Special  Projects 
Leading  Edge  Process  Consultants 
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Getting  Better,  Constantly  w 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS 


We  introduced  an  explosion  of 
new  classroom  ideas  this  fall... 


MINI  CMMI 


How  are  these  used  in  class?  Give  us  $1190  and  three  days  of  your  time,  and  you  can  find  out! 
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Getting  Better,  Constantly  (2) 

...which  we  piloted  in  class  before  becoming 
part  of  our  standard  process 


^eep  ill 

Don't  care 

Lose  Itl 

^CrsJ 

Improve  it., 

Afternoon  1  energizer  (student- ted 
■T.M.C.A,") 


92.9%  (13J 


7.1%  (1) 


0.0%  (0) 


0.0%  (0) 


^tion 


,  After 
:  pfayer”  r 


S.  ril  probably  keep  the  EINSTEIN,  but  replace  him  with  a  bobble-head  that's  larger  and  looks  much  cooler!  Create  Chart  Download 
I  also  may  not  unveil  him  until  Day  2,  and  111  emphasize  him  being  passed  on  whenever  students  provide  great  insights  or  EXAMPLES- 

Response  Response  | 
Percent  Count 


Keep  the  "Einstein"  concept;  go 
ahead  and  try  your  changes, 

I  appreciate  you  asking  me  Bill,  but  I 
really  don't  care. 

Actually,  Bill,  you  should  lose  him 
entirely, 


85.7% 


14.3% 


0.0% 


12 


answered  question 
skipped  question 
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introduction 


Response 

Count 


recot 


TlMPORTANTLV.Wouldyou 

coUeague? 


Response 

percent 


gpiNALLVAND 

training  to  a  trier 


Hide-eP'i«  comments- 


sntertaining 


jsefui  tidbits 


,^rtan....end 


insights- 


Measuring  Our  Training  Delivery  Results  dl 


PROCESS 
CIOIMSLJLT/VIMTS  - 


WHAT  WE  LOOK  AT 

•  Standard  SEI  Class  Evaluations,  aggregated  for  each  class 

•  Our  own,  more  customized  web  survey  (using  Survey  Monkey) 


MOST  IMPORTANT  QUESTION 

Would  you  recommend  our 
version  of  the  Introduction 
to  CM  Ml  training  to  a  friend 
or  colleague? 


INITIAL  RESULTS 

Since  doing  this  for  3 
classes,  everybody  has 
replied  “yes.” 
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PROCESS 

CO  ISI S  LJ  l_T/\  IM  TS  LL-C 


“Got  to  admit,  it’s  getting  better”  [Lennon,  McCartney] 

The  Future 


2010  and  Beyond 

@  Use  our  2009  gains  as  a 
foothold  for  continued 
improvement 

®  Keep  a  watchful  eye  on  the 
competition 

®  Add  more  structure  to  our 
process  improvement 
program 

•  Still  grounded  in  business 
value 

•  More  proactive,  a  bit  less 
“just-in-time” 
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Planning  Our  Improvements 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


•  Another  planned  improvement  cycle  in  early  2010 

•  How  to  find  the  time? 

•  “Skipping”  a  public  class  on  our  calendar 

•  Harsh  business  reality: 

•  Sometimes  you  need  to  make  /ess  money  now  so  you  can 
make  more  money  later 


Date 

Course 

Location 

Dec  8-10,  2009 

Introduction  to  CMMI  v1.2  (CMMI-DEV) 

Reston,  VA 

Mar  9-11,  2010 

Introduction  to  CMMI  v1.2  (CMMI-DEV) 

Reston,  VA 

Mar  12,  2010 

Services  Supplement  for  CMMI  v1 .2 

Reston,  VA 

April  13-15,  2010 

Introduction  to  CMMI  v1.2  (CMMI-DEV) 

Reston,  VA 
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A  Sampling  of  Future 
Improvements  w 
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Issue 


capacityand 

availability 


ImcroveiTient  I  Business  Reason 


Received  several  /Formally  tracf 
requests  for  private  /  requests  for 
training  in  Aug-Oct  that!  services  other 


we  lacked  the 
availability  to  handle. 
Some  consulting/ 
appraisal  requests 
also.  But  how  many? 
What’s  the  business 
value  of  opportunities 
missed? 


than  public 
training. 


If  indicated  by  dementi  (e.g., 
missed  oppo|luJ3jJi^,  increase 
income  bj(expandj^or  simply 
raising  prices?  PerHaps  reduce 
expenses  by  decreasing 
marketing  budget. 


V 


(SW 


Consider  new  servjc^  if  we 
doirt  oner  what  se^ral  people 
are  asking  for. 


strategic 


SERVICE 

management 

(STSM) 
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A  Sampling  of  Future 
Improvements  (2) 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


Issue 


Improvement 


Business  Reason 


► 


What  if  our  instructor  Preemptively  take  A  public  training  class  is  a 

gets  sick  and  can’t  mega-doses  of  $30K+  revenue  event  -  tons  of 

deliver  a  public  class  Vitamin  C.  More  money  for  a  small  company 

for  which  30  students  realistically,  have  like  ours.  Refunding  this  money 

are  enrolled,  half  of  a  back-up  could  be  crippling  -  and  the 

whom  have  flown  into  instructor.  customer  dissatisfaction  hit 


occasion? 


could  be  severe. 


IRP  vs.  SCON:  Which  One? 

Though  some  might  say  this 
issue  is  about  IRP,  it’s  so 
potentially  serious  that  we 
believe  we’ll  benefit  more 
from  applying  the 
principles  of  SCON  to  it 
Hey  whatever  works! 
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Parting  Words  of  Wisdom  Experience 

In  Conclusion 


Conclusions 


®  We’ve  begun  applying  the  CMMI  for  Services 
to  our  Marketing  and  Training  Delivery 
processes. 

®  Our  process  improvement  initiative  is  solely 
about  business  value.  We  have  no  current 
plans  to  attain  a  Maturity  Level  rating. 

®  We’ve  achieved  a  significant  net  income  gain 
in  the  past  year.  We  unquestionably  attribute 
much  of  that  gain  to  our  adoption  of  key 
CMMI-SVC  principles. 

®  Given  our  success,  we’ll  continue  adopting  the 
CMMI  for  Services  through  2010. 


=  > 

PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 
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Recommendations 

for  the  Ultra  Small  Organization  (i) 

1 .  Don’t  Panic! 


j=  > 

PROCESS 
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Yes,  we  realize  the 
CMMI  can  seem 
overwhelming.  It 
doesn’t  have  to  be  that 
way,  though.  Relax  and 
take  a  deep  breath 
before  you  proceed... 
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Recommendations 

for  the  Ultra  Small  Organization  (2) 


lAmilina 

PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


2.  View  the  CMMI  as 
an  Encyclopedia  of 
Good  Stuff 

It’s  chock  full  of  good 
ideas.  Probably  too  many 
for  you.  Some  of  them 
will  quickly  benefit  your 
organization.  The  others? 
Ignore  them  for  now. 
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Recommendations 

for  the  Ultra  Small  Organization  (3) 


> 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


3.  Focus  on  Your  Pain 

Show  immediate  benefit 
by  using  an  iterative  --  or 
“agile”  --  process 
improvement  approach. 
(Need  a  detailed 
example?  Check  out  the 
CM  Ml  Survival  Guide.) 


cMiyir  ^ 

Survival 


)ust 

Enoogli 

Process 

lmprovetn'=»^ 


/ 


V 


Suzanne  Garc^ 

forerpora  try 
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Recommendations 

for  the  Ultra  Small  Organization  (4) 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


4.  Abandon  Your 
“Compliance”  Mindset 

It’s  nice  to  be  compliant 
but  it’s  way  nicer  to  make 
money.  Focus  on  using 
pieces  of  the  CM  Ml  to 
achieve  your  business 
goals.  Measure  your 
success  with  dollars,  not 
a  Maturity  Level. 
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Recommendations 

for  the  Ultra  Small  Organization  (5) 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


5.  Avoid  the  “Big  Bang” 
Approach  at  All  Costs! 


Seriously,  do  you  want  to 
spend  the  next  two  years 
documenting  processes? 
And  then  gathering 
evidence?  And  then 
sheliing  out  tens  of 
thousands  of  doilars  for  an 
appraisal?  And  stili  not 
know  whether  you  truly  got 
better?  Um,  neither  do  I. 
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Recommendations 

for  the  Ultra  Small  Organization  (6) 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS  LL-C 


6.  Use  a  Just-in-Time 
Approach  to  Process 
Documentation 
Whenever  Possible 

You’ll  end  up  with 
process  descriptions 
more  rooted  in  reality, 
and  more  immediately 
useful. 
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Recommendations 

for  the  Ultra  Small  Organization  (7) 


PROCESS 

CIO  IM  S  LJ  LT/V IM  TS 


7.  At  Some  Point  You  May 
Need  to  Reconsider  the  First 
6  Recommendations 


Someday  you  may  no  longer  be 
“ultra  small.”  Your  informal 
communication  channels  may 
break  down,  and  the  written  word 
will  become  more  important.  You 
can’t  always  generate  process 
documents  “just-in-time.”  You  may 
need  to  demonstrate  compliance 
to  external  customers,  or  even 
yourself. 

Still,  you  may  never  want  to 
abandon  Recommendation  1. 
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My  Other  Presentations  This  Week 

®  An  Overview  of  CMMI-SVC  for 
CMMI-DEV  Enthusiasts 

•  Wednesday  11/18 

•  8:45-9:15  a.m. 

•  Wind  Star  Room 


j=  > 

PROCESS 
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®  CMMI  in  the  Social  Media 
(for  the  Social  Media-Challenged!) 

•  Wednesday  11/18 

•  10:45-11:30  a.m. 

•  Grand  Mesa  F 
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Any  Questions? 


PROCESS 
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Website:  www.CmmiTraining.com 
Blog:  www.CmmiForServicesDiary.com 
Twitter:  CmmiRox 
Linkedin:  www.linkedin.com/in/ 
billsmithleadingedge 


Upcoming 
Public  Classes 

In  Reston,  VA 
(DC  Metro  Area) 

SEI  Introduction  to  CMMI 

•  Dec  8-10,  2009 

•  Mar  9-11,  2010 

•  Apr  13-15,  2010 

SEI  Services  Supplement 
for  CMMI  (CMMI-SVC) 

•  Mar  12,  2010 
Private  Classes? 

•  Bill@CmmiTraining.com 
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naytheon 

Customer  Success  Is  Our  Mission 


Directive  Documents 
and  ITAR  Made  Easy 


Ken  Weinberg 

El  Segundo,  CA 
kiweinberg@raytheon.com 

November  17,  2009 


Copyright  ©  2009  Raytheon  Company.  All  rights  reserved. 
Customer  Success  Is  Our  Mission  is  a  trademark  of  Raytheon  Company. 


Overview 


Raytheon 

Space  and  Airborne  Systems 


•  What  Prompted  a  Review  of  This  Process? 

•  WhatisITAR? 

•  What  other  Export  Regulations  Apply? 

•  How  is  sensitive  data  protected? 

•  What  was  the  Initial  Review  Process? 

•  What  types  of  Directive  Documents  contain 
restricted  data? 

•  Analyzing  the  Sensitive  Documents 

•  New  Review  Process 

•  Benefits 

•  Lessons  Learned 


Page  2 


What  Prompted  a  Review  of  This 
Process? 


Raytheon 

Space  and  Airborne  Systems 


*  The  ITAR  Review  of  Documents  Occurs  Once  the 
Content  is  Finaiized,  Prior  to  Reiease 

*  The  Quantity  of  Documents  varies  by  monthiy  reiease 

*  Reviewers 

-  Specific  training 

-  Core  Team  each  Month 

-  Extra  Reviewers  During  Peak  Periods  supplement  the  core  team 

*  One  of  the  Outside  reviewers  found  the  process  tedious 
and  expiored  methods  of  improvement 
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What  is  ITAR? 


Raytheon 

Space  and  Airborne  Systems 


*  International  Traffic  in  Arms  Regulations  (ITAR) 

-  Government  regulations  that  control  the  export  and  import  of 
defense-related  articles  and  services  on  the  United  States 
Munitions  List 

-  Implements  the  provisions  of  the  Arms  Export  Control  Act 

•  Information  required  for  the  development, 
manufacturing,  operation,  etc.,  of  defense  articles 


Goal  is  to  safeguard  US  national  security  and 
further  US  foreign  policy  objectives 
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What  other  Export  Regulations 
Apply? 


Raytheon 

Space  and  Airborne  Systems 


•  Export  Administration  Regulations  (EAR) 

•  Contains  the  Commerce  Control  List  (CCL)  of  regulated 
commercial  items,  including  “dual-use”  items  that  have 
commercial,  military  or  proliferation  applications. 

•  Broad  array  of  commodities,  software  and  technologies 
including, 

-  Building  materials 

-  Circuit  boards 

-  Automotive  parts 

-  Blue  prints 

-  Design  plans 

-  Retail  software  packages 

-  Technical  information 
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Not  Export  Sensitive 


Raytheon 

Space  and  Airborne  Systems 


•  Information  related  to  general  scientific,  mathematical  or 
engineering  principles  that  is  commonly  taught  in 
schools  and  colleges 

•  Information  that  is  in  the  public  domain 

-  MIL  Standards 

•  General  marketing  information  or  basic  system 
descriptions 

•  “Form,  Fit  and  Function”  Descriptive  Information 


Not  needed  to  safeguard  US  national  security 
and  further  US  foreign  policy  objectives 
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“EXPORT” 


Raytheon 

Space  and  Airborne  Systems 


“Export”  includes  not  only  the  shipment  of 
products  abroad,  but  also  release  of  technical 
data  to  a  foreign  person  which  is  deemed  an 
export  by  its  mere  disclosure  or  transfer  to  a 
foreign  person,  even  if  within  U.S.  borders. 
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Space  and  Airborne  Systems 


How  is  sensitive  data  protected? 

•  Marked  as  “Export  Sensitive”: 

This  document  (or  software  if  applicable)  contains  data  whose 
export/transfer/disclosure  is  restricted  by  U.S.  law.  Dissemination  to 
non-U. S.  persons  whether  in  the  United  States  or  abroad  requires 
an  export  license  or  other  authorization. 

•  Screen  displays  when  document  is  selected: 


“WARNING  -  The  document  you  are  attempting  to  view  may  contain 
Technical  Data  within  the  definition  of  the  International  Traffic  in 
Regulations  (ITAR)  and  is  subject  to  the  export  control  laws  of  the 
U.S.  Government.  Transfer  of  this  data  by  any  means  to  a  Foreign 
Person,  whether  in  the  United  States  or  abroad,  without  an  export 
license  or  other  approval  from  the  U.S.  Department  of  State,  is  strictly 
prohibited.” 
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How  is  sensitive  data  protected? 
(continued) 


Raytheon 

Space  and  Airborne  Systems 


•  The  “Filter”  that  permits  access  when  an  ITAR  document  is  selected: 


RayHieon 


A 

Simplified  Sign  On 


Warning:  After  you  login,  you  may  see  the  message:  "You  are 
being  redirected  to  a  connection  that  is  not  secure. 

Please  Click  OK.  This  message  appears  when  you  leave 
SSO,  which  is  encrypted,  and  go  to  the  w-eb  site  you 
requested,  which  is  not  encrypted. 


Login  Help 


What  is  my  employee  number? 

Your  employee  number  is  listed  as  part  of  your  Raytheon 
Directory  Services  entry.  If  you  are  unsure  of  the  number 
please  search  for  your  entry  in  Directory  Services.  Your 
employee  number  is  listed  in  your  entry'. 

What  is  my  password? 

Your  password  is  your  Raytheon  Directory  Services 
password.  To  change  your  password,  please  go  to  the 
Direiitorv  Services  Change  Mv  Password  self-service  web 
site.  You  must  have  a  valid  email  account  in  Directory 
Services,  and  answer  your  Security  Questions  correctly  to 
have  your  password  reset. 

If  you  need  further  assistance  to  reset  your  password, 
please  contact  the  Raytheon  Help  Desk  at  1.877.844.4712 
[international  employees  only  call  1.469.995.2911;. 

The  Corporate  password  reset  process  is  located  on  the 

Enterprise  Security  Services  web  site  at: 

l~itti:» :  s  0  CLi  I'it . .  it.  la  r.  co  m  •  ii'  ro  ce  cl  u  re  s  'v  ass  vvo  re!  re  s  ets .  htni  I 
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Employee  Database 


Raytheon 

Space  and  Airborne  Systems 


Raytheon 

Employee  Profile 

Personal 


Name:  Ken  Weinberg 
Full  Name:  Kenneth  I  Weinberg 


iliqible  to  access  US  export  ^ 

- : — „  ■  Yes  -  Allowed 

controlled  data: 

irt  Control  Awareness  Training:  Yes_ 


Contact 


Mail 


Phone  Number:  310-647-2669 
Fax  Number:  310.647.2263 

Email  Address:  kiweinberq@ravtheon.com 
Work  Mail  Drop:  EO/E01/A173 
Work  Mailing  Address:  2000  E.  El  Segundo  Blvd 
City,  State  Zip:  El  Segundo,  CA  90245 
Work  PO  Box  Address:  PO  Box  902 

City,  State.  Zip:  El  Segundo,  CA  90245 
City:  El  Segundo 
State:  CA 


As  of  Date:  2009-10-27 
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Raytheon 

What  was  the  Initial  Review  Process? 

•  T rained  reviewers  review  each  paragraph  of  each 
document  to  determine  status 

-  Not  Export  Restricted  -  Contains  No  Technical  Data  or  Technical 
Data  which  is  not  sensitive 

-  Export  Restricted  -  Contains  Technical  Data  which  is  Sensitive 

•  Review  Expense  must  be  Included  in  Release  Costs 

•  Review  time  must  be  Included  in  Release  Schedule 

•  Review  must  be  thorough  to  ensure  not  a  threat  to 
national  security 
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What  types  of  Directive  Documents 
contain  restricted  data? 


Raytheon 
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•  237  Directive  Documents  -  2  are  restricted  (<1%) 

•  361  Non-Directive  Enablers  -  27  are  restricted  (7%) 


•  Directive  Documents  tell  “What”  to  do  and  do  not  tell 
“how”  to  design  or  manufacture 

•  Non-Directive  Enablers  are  guidelines,  templates, 
samples,  etc.,  which  often  tell  “how”  to  design  or 
manufacture 
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Which  Documents  are  Export 
Controlled? 
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Purchased  Parts  Documentation  - 

Engineering  Review  Board  Template  - 

Parts,  Materials  and  Processes  Control  - 

Plans 

Managing  GIDEPs,  General  Alerts,  Supplier- 
Nonconforming  Notices 

Built-In  Test  Checklist  - 

Prohibited  Materials  Program  - 

Separate  Notes  and  Parts  Listing  - 

Robust  Design  Guide  - 

Simulation  Development  - 

Connector  Saver  Usage  - 

Onboard  Regulator  Power  &  Efficiency  - 

Calculations  for  Two  Phase  Buck  Converter  _ 

Current  Sense  Amplifier  Design  Guide 
Wilkinson  Divider  - 


Rat  Race  Coupler 
Calculator  for  HW  Design 

Transceiver  Performance  Worksheet  for 
HW  Analysis 

RFMW_SPW  Worksheet  for  Transceiver 
HW  Analysis 

RRFC  Foundry  Website/Design  Rules 
Fastener  Design  Guide 
Structural  Guidelines 
Cable  Design  Guidelines 
Packaging  Module  Design 
Multi-Layered  Insulation  Specification 
Connector  Mating  Interface  Design 

Inspection  of  Planar  Waveguides  for  High- 
Power  Lasers 

Real-Time  Laser  Beam  Analyzer 

X-Ray  Fluorescence  Equipment  for 
Prohibited  Materials  Scanning 


AM  Export  Sensitive  Documents  are  Technicai 
and  from  Engineering  Discipiines 


New  Review  Process 


Raytheon 

Space  and  Airborne  Systems 


•  Documents  that  are  free  of  technical  content  do  not 
need  a  trained  ITAR  review 

-  Developed  Guidelines  Regarding  Technical  Content 

-  Author  or  Discipline  Process  Representative  Certifies  that  there  is 
no  technical  content  in  document 

-  Used  for  disciplines  which  are  not  typically  technical  (e.g., 
Configuration  Management)  and  have  no  history  of  ITAR  sensitive 
documents 

•  What  is  Reviewed  by  Trained  Reviewers 

-  Directive  Documents  -  Only  sections  5  (Instruction) 

-  Non-Directive  Enablers  -  Only  section  4  (Technical  Content) 

-  Author  or  Discipline  Process  Representative  Certifies  that  there  is 
no  technical  content  in  remaining  sections 
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Technical  Content  Guidelines 


Raytheon 

Space  and  Airborne  Systems 


•  Technical  Content  is  technology  associated  with  a 
product 

-  Instructions  to  conformal  coat  electronics 

-  Geometric  Tolerance  Guidelines  for  a  specific  item 

-  How  to  measure  timing  of  Image  Processing  Software 

•  Administrative  data,  Business  Data,  Instructions  without 
the  mention  of  product  technology  are  not  technical 

-  Peer  Review  Process 

-  Earned  Value  Calculations 

-  Cost  Estimation/  Cost  Collection 
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New  Review  Process  Flow 


Raytheon 

Space  and  Airborne  Systems 


Benefits 


Raytheon 

Space  and  Airborne  Systems 


•  New  Method  Has  Lower  Cost 

•  New  Method  Has  Shorter  Schedule 

•  Little  Risk  of  Release  of  Sensitive  Material 

-  Stored  on  internal  servers 

-  All  non-marked  material  requires  review  before  delivery  outside  of 
Raytheon 


Review  is  Still  Thorough  Enough  to 
Ensure  no  Threat  to  National  Security 
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Lessons  Learned 


Raytheon 

Space  and  Airborne  Systems 


•  Analysis  of  the  Data  of  a  “Working”  Process  Can 
Provide  Significant  improvement 

*  “Outsiders”  looking  at  a  process  can  often 
recognize  areas  for  improvement  better  than  those 
intimateiy  involved  in  the  process. 
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Raytheon 

Space  and  Airborne  Systems 


Questions  ?  ?  ? 


IFechsoft 

'Technical  Software  Services,  Inc. 

CMMI®  in  a  Small  Company:  The 
Cobbler's  Children  Can  Have 
Shoes  (and  best  practices) 


Michael!  Knox 

Technical  Software  Services,  Inc. 
Director  of  Process  Engineering 
SEI  Authorized  Instructor 


Cara  E.  Smith 

Technical  Software  Services,  Inc. 
Deputy  Director  of  Process 
Engineering 
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ECHSOFT 

'Technical  Software  Services,  Inc. 


Overview 


•  Once  upon  a  time 

•  Making  shoes 

•  Tradecraft  secrets 

•  Morals  of  the  story 


Many  of  us  are  like  the  cobbler. 

Small  businesses  focused  on  servicing  our  customers, 
improving  their  organizations,  and  making  them  look  good. 

At  some  point,  we  have  to  take  care  of  ourselves  too! 


SEI  PaHner 


( Mr  I  lor 


ECHSOFT 

'Technical  Software  Services,  Inc. 


(lariK'jfie.Mfllon. 


r  SEI  Partner 


dprECHsoFT  Onc0  uDon  a  tim0 

^^P-Technical  Software  Services,  Inc.  I  I  I  I  W I  I  I  I  •  •  • 

...a  small  company  called  TECHSOFT  was  formed 

•  Technical  Software  Services  (TECHSOFT)  Inc., 
founded  in  1990  in  Pensacola,  Florida 

•  Started  as  purely  software  development  and 
maintenance  company 

•  Evolved  over  19  years  into  a  true  IT  company  and 
now  provides: 

Systems  engineering 
Software  engineering 
Security  engineering 
Process  engineering 
^  Network  services 

Training  services  (web-based  development) 


SEI  PaHner 


( Mrllor 


dprECHsoFT  Onc0  uDon  a  tim0 

^^P-Technical  Software  Services,  Inc.  I  I  I  I  W I  I  I  I  •  •  • 

...it  was  comprised  of 

•  Two  offices  -  Pensacola,  FL;  Charleston,  SC 

•  40  employees  with  diversified  backgrounds: 

^  U.S.  Navy  communications  and  National  Security 
Agency  (NSA)  computer  security  backgrounds 

^  Serve  as  certified  adjunct  faculty  at  local 
universities 

Hold  a  broad  range  of  Certifications  and  Technology 
Competencies 

^  Work  on  ISO/IEC/IEEE  systems,  software,  services, 
and  project  management  standards 


SEI  PaHner 
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'Technical  Software  Services,  Inc. 


Once  upon  a  time... 

...and  it  worked  for 


Primarily  Department  of  Defense: 


USMC 


cc 

Corry 

Station 


NCTAMS  LANT  NETC  NSGA 

DET  Pensacola  Pensacola 


SF/mR 

_ * 

V 

Systo/ns  Cantof 
ATLAfi/rtC 


GENERAL  DYNAMICS 

Ordnance  and  Tactical  Systems 

Niceville,  FL 


(  CENTURUM 


Charleston,  SC 


Also  commercial  banks,  law  firms,  health  care, 
universities/colleges: 


Memphis  &  Shelby  Co  University  of 

Tennessee  W  Florida 


Pensacola 
Junior  College 


Santa  Rosa  Co. 
Schools 


SEI  Partner 
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'Technical  Software  Services,  Inc. 


ECHSOFT 

'Technical  Software  Services,  Inc. 


Making  shoes... 

...for  Others 


•  Early  1990s  -  2001 

Developed  software  and  security  engineering  solutions 

^  Implemented  the  SW-CMM  at  several  NSA  elements  and  at  a 
U.  S.  Navy  Worldwide  Software  Support  Activity  (SSA)  in 
Pensacola 

«  Focus  was  software  and  later  web-based  training 
development 

•  2001-2002 

Engaged  by  SPAWAR  Systems  Center  Charleston  (over  $2B  a 
year  in  revenue)  to  help  them  achieve  their  goal  of 
becoming  a  World  Class  Systems  Engineering  organization 

«  Opted  to  implement  the  new  CMMT  model  instead  of  the 
SW-CMM 

SEI  Pa  liner 
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'Technical  Software  Services,  Inc. 


Making  shoes... 

...for  Others 


•  2002-2009 

Successfully  led  organizations  to  CMMI®  ML2  and 
MLS: 

«  SSC-C  (ML2  in  2005)  (ML3  in  2007) 

^  Implemented  systems/software  processes,  training, 
and  established  Process  Improvement  infrastructure 

General  Dynamics  Ordnance  and  Tactical  Systems, 
Niceville  FL  (ML2  in  2007)  (ML3  in  2009) 

^  Centurum  Inc.,  Charleston  SC  (ML2  in  2009) 

«  SSC  Atlantic  -  New  Orleans  {ML3  in  2009) 

^  Committed  funding  and  people  to  CMMI®: 

^  3  authorized  SCAMPP“  Lead  Appraisers  on  staff 
«  3  authorized  CMMI®  Instructors  on  staff 


SEI  PaHner 


( Mrllor 
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'Technical  Software  Services,  Inc. 


Making  shoes... 

...for  ourselves? 


•  TECHSOFT  was  such  a  busy  cobbler  making  shoes 
for  his  customers  that  he  had  no  time  to  make  some 
for  his  own  children 

^  Why  not? 

^  Customer  work  kept  us  very  busy 

^  Pursuing  CMMI®  ML3  for  TECHSOFT  would  be  an  overhead  cost, 
and  small  companies  cringe  at  the  word  "overhead" 

^  Concerned  with  the  ROI  -  would  ML3  help  TECHSOFT  win 
contracts  against  Boeing,  Lockheed,  Raytheon? 

^  Employees  are  already  wearing  multiple  hats 

^  Small  projects 

«  Limited  tools 

•  The  "Head  Cobbler"  decreed  that  it  was  time  for 
TECHSOFT  to  practice  what  we  were  preaching 

Formally  started  our  journey  to  CMIVir  ML3 


SEI  Paflner  Cin  iM^iJrirVlrllon 
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'Technical  Software  Services,  Inc. 


Making  shoes... 

...was  the  right  decision 


•  Why  did  we  do  it? 

^  It  was  the  right  thing  to  do 

It  would  give  TECHSOFT  a  competitive  edge  over 
other  small  businesses 

^  Contract  wording  was  changing  to  include 
achievement  of  CMMI®  ML3 

We  had  seen  the  benefits  of  our  customers 
implementing  CMMI® 

^  We  knew  HOW  to  do  it  and  were  using  many  of  the 
best  practices 

^  If  we  did  not  do  it  now,  we  would  never  do  it 


SEI  PaHner 
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'Technical  Software  Services,  Inc. 
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'Technical  Software  Services,  Inc. 


ECHSOFT 


Tradecraft  secrets... 

...our  Strategy 


®  Goal 

«  Achieve  CMMI®  MLS 

•  Approach 

^  Roadmap:  Developed  a  Process  Improvement  Plan 
^  Share:  Developed  corporate  plans  for  common  areas: 

^  Quality  Assurance,  Requirements  Management,  Risk  Management, 
Configuration  Management,  Decision  Analysis  and  Resolution,  and 
Supplier  Agreement  Management 

«  Projects  responsible  for  Project  Management  and 
Engineering  Plans 

«  Set  up  monthly  status  meetings 

^  Set  up  progress  benchmarks  in  the  form  of  Class  C  and  B 
assessments 
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^^P-Technical  Software  Services,  Inc.  I  I  I  I  W  I  •  •  • 

...our  infrastructure 

•  Formally  established  the  PI  infrastructure: 

^  Management  steering  Group  (MSG) 

Engineering  Process  Group  (EPG) 

Mentors 

Evidence  Custodian 
^  Internal  Assessors 

•  Designated  the  Appraisal  WIZARD®  as  the 
evidence  repository 

Evidence  Custodian  was  responsible  for  entry  of  all 
data  into  the  Appraisal  WIZARD®  and  maintenance 
of  the  evidence  files 


SEI  PaHner 
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ECHSOFT 


Tradecraft  secrets... 


...our  infrastructure 


•  MSG  -  Department  Heads  (4) 

•  EPG  -  Department  Lead  Systems  Engineers  (4) 

•  Mentors 

Designated  three  experienced  individuals  as  PI  Mentors  to  lead 
the  effort  for  3  projects 

Assigned  to  populate  data  for  OPD,  OPF,  and  OTP 
Provided  mentoring  and  coaching  regarding  evidence  required 
^  Drafted  documents  as  needed 

^  Conducted  monthly  status  meetings  with  Focus  projects-  senior 
management  involved 

«  Identified  and  tracked  action  items  to  closure 

•  Internal  Assessors 

Identified  two  other  CMMI-experienced  employees  to  conduct 
multiple  Class  C  assessments  and  then  a  Class  B  assessment 


SEI  Paflner 
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^^P-Technical  Software  Services,  Inc.  I  I  I  I  W  I  •  •  • 

...projects'  reaction 

•  Just  like  the  projects  at  other  organizations!! 

Slow  in  getting  started 

Not  happy  about  writing  plans 

Struggled  with  collecting/centralizing  evidence 

«  Assumed  the  PI  Mentors  would  do  EVERYTHING 
and  they  could  be  bystanders 

Claimed  that  they  had  a  REAL  JOB  to  do  for  some 
customer 


SEI  PaHner 
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ECHSOFT 


Tradecraft  secrets... 

...what  happened 


•  Early  2007  -  Initial  schedule  slipped  by  3  months 

Resource  constraints/conflicts  are  real  issues 

•  Mid  2007  -  Established/Updated  Standard  Processes,  Process 
Manuals,  and  Infrastructure 

•  Mentoring  Projects  -  documenting  project  processes  and  SOPs 

•  Initial  Class  B/C  project  assessments  (July,  2008)  results  were 
poor!  Too  many  gaps 

Schedule  slipped  another  3  months  to  plug  gaps 

•  Nov,  2008  -  Firm  dates  and  Lead  Appraiser  selected 
»  Full  Class  B- Jan,  2009 

•  Final  evidence  collection 

•  SCAMPI  A  Appraisal  held  in  May  2009 

TECHSOFT  achieved  CMMI*  MLS 


SEI  Paflner 


( Mrllor 


(ISirECHSOFT 

^^►Technical  Software  Services,  Inc. 


.  SEI  Partner  Curm-gieMfllon 


I 


ooo 


dprECHsoFT  Moral  of  the  storv 

^^P-Technical  Software  Services,  Inc.  I  V  I  I  I  I  W I  I  W  I  V  •  •  • 

...meeting  the  schedule 

•  The  project  evidence  collection  effort  ALWAYS 
starts  slowly 

Recommendation:  Conduct  a  Class  C  within  60  days  of 
commencing  the  effort.  Schedule  several  Class  C's  and  a 
Class  B 

•  Only  with  a  hard,  firm  date  for  the  SCAMPI  will  the 
projects  start  moving  faster 

^  Recommendation:  Build  a  realistic  schedule  with 
multiple  Class  C  assessments/appraisals  and  then  a 
Class  B 


Red  and  Yellow  results  (weaknesses)  from 
Class  C/B  assessments  get  attention 
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...identifying  and  storing  evidence 

•  The  project  personnel  had  good  knowledge  of 
processes  but  limited  knowledge  of  CMMI®  and 
needed  help  determining  which  piece  of  evidence 
went  with  which  practice 

Recommendation:  Identify  Mentors  early  on  to  help 
projects  with  collecting  evidence 

•  Needed  a  central  location  to  organize/map  to 
CMMI®  and  store  the  projects'  evidence 

Recommendation:  Invest  in  an  automated  tool  (e.g.: 
Appraisal  Wizard®)  to  collect/store  project  evidence  and 
be  used  for  the  Class  C/B  assessments,  SCAMPI  A 
Readiness  Review  and  On-site 


SEI  PaHner 
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'Technical  Software  Services,  Inc. 


ECHSOFT 


Moral  of  the  story.. 


involvement  of  'Head  Cobbler 


•  If  left  alone,  the  project  personnel  would  often  fall 
behind  schedule 

Recommendation:  Hold  frequent  progress  meetings 
between  mentors,  project  personnel  and  invite  "Head 
Cobbler"  for  additional  effect 

•  With  Senior  Management  support  made  known  to 
all,  personnel  who  aren't  completely  on-board 
with  CMMI®  may  continue  to  complain,  but  will 
still  do  their  part 

^  Recommendation:  Make  sure  everyone  in  the  company 
knows  that  Senior  Management  is  on-board  with  the 
decision.  Have  Senior  Management  go  directly  to 
Project  Leaders  and  Team  Members,  and  not  leave  the 
burden  to  the  Mentor(s) 


dprECHsoFT  Moral  of  the  storv 
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...committed  project  resources 

•  Small  Organizations  need  to  take  advantage  of 
everyone's  skills  to  share  the  roles 

Project  Manager  -  involved  and  committed  to  success 

Document  specialist/Technical  Writer  role  for 
coordinating  documentation,  revisions 

Active,  skilled  PPQA  manager  is  a  great  benefit 

Can  also  serve  as  the  Measurement  Analyst 

Useful  plans  are  built  by  the  key  players;  shelfware  is 
built  by  the  novice  or  new  contractor 

Don't  let  one  person  wear  too  many  hats 

New  technology  and  complex  systems  are  NOT 
necessary  for  success 
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...be  creative  with  model 

•  Small  organizations/projects  can  struggle  with 
certain  areas  (DAR,  SAM) 

Recommendation:  Look  for  creative  ways  within 
organization  to  adapt  /  tailor  the  model 

SAM  -  "free"  software  contains  license  agreements;  corporate 
purchases  of  equipment 

DAR  -  Organization-wide  software  tools;  Charleston  office 
decisions/alternatives 

•  Shorter  duration  projects/tasks  are  difficult  to 
include  in  assessment  planning 

«  Recommendation:  Consolidate  similar  projects  and  on¬ 
going  tasks  into  longer  term  programs  for  assessment 

«  Utilize  evidence  from  multiple  tasks  to  ensure  full  life  cycle 
coverage 
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Moral  of  the  story... 

...benefits 


•  What  has  success  meant? 

New  people  hired  in  the  last  year  are  up  to  speed 
quickly  due  to  the  developed  processes 

Improved  our  overall  training  in  the  processes  and  sub¬ 
processes 

Morale  has  improved 

Measurement  is  more  accurate 

More  sharing  of  ideas  and  processes 

Have  institutionalized  process  reviews 

Now  have  a  formal  infrastructure  for  Process 
Improvement 

«  MSG/EPG  now  meet  regularly  every  quarterly,  thus 
resulting  in  improved  processes 
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...it  was  all  worth  it 

•  Took  awhile  to  gain  full  commitment 

•  Used  the  same  approach  for  TECHSOFT  that  we 
used  for  clients 

•  Don't  be  surprised  by  same  hurdles  faced  with 
clients 


It  was  all  worth  it!!!!! 
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Any  Questions? 


Michael!  Knox 

Technical  Software  Services,  Inc. 
Director  of  Process  Engineering 
SEI  Authorized  Instructor 


Cara  E.  Smith 

Technical  Software  Services,  Inc. 
Deputy  Director  of  Process 
Engineering 


United  Space  Alliance 


Overview 


United  Space  Alliance  (USA)  Launch  Processing  System 
Software  Development  organization  received  a  CMMI-DEV 
+  IPPD  Level  3  rating  in  September  2009 

■  Employed  a  lean  approach  to  appraisal  activities  resulting  in  >50%  cost 
and  schedule  reduction 

■  Proved  that  appraisals  can  be  done  faster,  better,  cheaper 

Focused  -  Innovative  -  Trailblazers 

This  presentation  provides: 

■  Company  CMM/CMMI  history  and  background 

■  Objectives,  challenges  and  results  of  the  recent  CMMI  appraisal 

■  Methodology  and  examples  of  lean  appraisal  practices 

■  Advice  for  others  wishing  to  embark  on  a  similar  journey 

US^ 


United  Space  Alliance 


Who  We  Are  . . . 

United  Space  Alliance 


History 


□  2002--USA  began  its  journey  towards  CMM  Level  3. 

□  2003--A  mini-assessment  was  conducted  across  USA  elements  to  determine 
readiness  for  a  CMM  Level  3 

■  A  common  software  process  and  appraisal  at  the  company  level  was  deemed  not 
achievable 

■  Decision  was  made  for  each  element  to  develop  their  own  framework  and 
conduct  individual  assessments 

□  2004-LPS  Software  Development  achieved  SW-CMM  Maturity  Level  3 

□  2006-LPS  Software  Development  completed  CMMI-DEV  (v  1.1)  Maturity  Level  3 

■  No  prior  CMMI  experience 

■  Pathfinder  for  the  entire  company 

■  All  of  the  other  business  units  benefited  from  the  knowledge  and  expertise  gained 
by  LPS  Software  Development 

□  2009 — LPS  Software  Development  completed  CMMI-DEV+IPPD  (v  1.2)  Re-Appraisal 
Maturity  Level  3 

■  LPS  Software  Development  organization  was  the  pathfinder  for  the  entire 
company  in  re-appraisal  activities 


USA 
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United  Space  Alliance 


CMMI  or 
Bust!  ■ 


Background 


□  Demonstrated  compliance  with  CMMI-DEV  v1.1  Maturity  Level  3  in  March 


2006 


□  Business  decision  was  made  to  forego  any  further  appraisal  activities 

■  CMMI  rating  expired  in  March  2009 

□  Business  shift  with  the  possibility  of  Shuttle  Program  extension  and  the 
need  for  a  current  CMMI  v1 .2  rating  in  order  to  bid  on  future  contracts 

■  Decision  for  LPS  Software  to  conduct  a  CMMI  v1 .2  re-appraisal  (early 
April  2009) 


USA 
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United  Space  Alliance 


Why  we  did  it 


Main  objectives  of  the  re-appraisai: 


□  Ensure  the  software  development  process  remains  compliant  with 

■  Shuttle  customer  requirements  (NSTS) 

■  CMMI-DEV  Maturity  Level  3  framework 

□  Ensure  the  LPS  Software  Development  processes  meet  the 
customer  requirements  for  the  Constellation  Program  in  preparation 
for  future  work 

□  Compliance  with  version  1 .2  of  the  CMMI-DEV  model 

□  Enhance  the  software  development  framework  to 

■  Improve  and  refine  the  processes 

■  Ensure  continued  improvement  in  the  quality  and  reliability  of  delivered  products 


USA 


United  Space  Alliance 


The  Road  Ahead 


Challenges 


□  Sense  of  urgency  with  the  pending  release  of  the  Exploration  Ground 
Launch  Services  (EGLS)  Request  for  Proposal  (RFP)  for  the 
Constellation  Program 

□  Concern  from  NASA  with  the  amount  of  time  invested  for  appraisal 
activities  versus  contractual  obligations  and  value  add  for  the  customer 

□  Lack  of  work  during  transition  from  Shuttle  to  new  Constellation  program 
for  re-appraisal  activities 

□  LPS  Software  Development  was  challenged  to  conduct  the  re-appraisal 


1 .  Under  $150K  for  external  Lead  Appraiser  services  (paid  for  by  the 
company) 

2.  $125K  for  appraisal  team  members  (paid  for  by  the  company) 

3.  PHD  preparation  by  project  personnel  at  an  effort  of  1680  labor  hours 
(paid  for  by  Shuttle  Program). 

4.  Schedule  challenges... calendar  year,  before  RFP — moving  target 


USA 
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United  Space  Alliance 


Results-Cost  Savings 


Overhead  Cost  Assessment 


Lean  Re-Appraisal  Approach 

□  Less  training  required  (experienced  team) 

□  Removal  of  Class  B 

□  Condensed  Readiness  Review 

□  Condensed  SCAMPI  A 

□  *PIID  implementation 


Resulting  in 


□  Reduced  Lead  Appraiser  cost  by  54% 

□  Reduced  SCAMPI  activity  cost  by  63% 


A 


Met  Challenges  1  &  2 


United  Space  Alliance 


*See  next  slide 


Lean  Re-Appraisal  Approach 


Re-Appraisal  PHD  Development  Bfort 


□  Reused  PHD  format  with  minimal  changes 

□  Reused  Model  interpretation  of  required  OE 

□  Experienced  PHD  team  members 


1600 

1200 

800 

400 


Resulting  in 

□  Reduced  PHD  preparation  activities  by  56% 


0 


Met  Challenge  3 


USA 


United  Space  Alliance 
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Results-Schedule 


2006  Appraisal  Timeline 


2009  Re-Appraisal  Timeline 


United  Space  Alliance 


United  Space  Alliance 


Lean  Methodology 


Lean  Factors 

Appraisal 

Re-Appraisal 

How 

Team  Makeup 

8  Appraisal  Team  Members  (ATM) 

6  Appraisal  Team  Members  (ATM) 

Reduced  PHD  OE 

4  ATMs  had  no  previous  experience 

AM  ATMs  had  either  PHD  or  CMMI  appraisal 
experience 

Leveraged  USA  ATM 
Experience 

Appraisal  Time 

5  Day  Readiness  Review  activity 

3  Day  Readiness  Review  activity 

Lean  Concept  Applied 

10  Day  SCAMPI  A 

8  Day  SCAMPI  A 

Experience  & 

Lean  Concepts 

PHD  Reuse 

New  PHD  format/tool 

Reused  general  PHD  format/tool 

Experience 

All  model  practices  had  to  be  interpreted  in  relation  to 
the  organization 

Practice  interpretations  were  reviewed  and  reused 
85%  of  the  time 

Leveraged  Previous  PIIDs 

Separate  objective  evidence  (OE)  for  project  and 
tasks 

Effective  techniques  for  project/task  OE  combinations 

Lean  Concept  Applied 

4  Projects  with  4  Focus  Tasks 

3  Focus  Projects  with  3+  Tasks 

Model  Interpretation 
Maturity  & 
Experience 

Training  methods 

PHD  workshop  used  canned  SEI  examples/formats 
activities 

PHD  workshop  used  previous  appraisal 
orgnaizational  PIIDs 

LA  Creative  Approach 

Appraisal  team  training  used  canned  SEI  training 
exercises 

Appraisal  team  training  used  current  PIIDs  for 
exercises 

USA 
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Reduction  of  required  PHD  evidence 

□  Artifact  reuse 

■  Replaced  2006  evidence  with  current  version  of  same  artifact. 

o  Estimate  85%  of  evidence  types  were  reused 

■  Reduced  unique  artifacts  by  37% 

□  Direct  evidence  reduced  by  22% 

□  Minimal  Indirect  evidence  provided 

■  Reduced  by  62% 

■  1  piece  of  evidence  per  project  per  goal 

Leveraging  interviews  for  objective  evidence 

□  Affirmations  were  required  for  model  coverage  (not  reiying  on 
indirect  evidence) 

□  LA  provided  generic  scripts  customized  for  organization. 

■  Scripted  questions  were  mapped  to  modei  practices 

■  Reduced  Appraisai  team  time  for  script  preparation  and  note 

tagging 


United  Space  Alliance 


Lean  Methodology^ 

PHD  Size  Assessment 


Resulting  in 

□  Reduced  number  of  PHD  cells  populated  by  39%  from  2006  to  2009 


Objective  Evidence  Comparison 


2009  PHD  Size  Planning 


USA 
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Lean  Methodology^ 

Appraisal  Activity  Assessment 


•  Decision  was  made  to  track  types 
appraisal  activities  using  USATS 

-  Appraisal  Planning 

•  Planning 

•  Tracking 

-  Schedule 

-  Status  Reporting 

-  CM  of  PHD  Artifacts 

-  Appraisal  Execution 
(internal  personnel  involved  in 
interview  and  meeting  support) 

-  Process  Compliance  Audits 

(PHD  Review  &  Development) 

•  By  Process  Area  (PA) 

-  SCAMPI  Activities 

•  Appraisal  Team  Training 

•  Readiness  Review 


USA 


SCAMPI  A 


May  June  July  Aug  Sept 


]  Panning  &  Tracking  — ■ — Appraisal  Activities  —  HID  Development 


United  Space  Alliance 
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Lean  Methodology- 


y  •  •f. 


¥- 


v'iVfj 


■^'  '  ‘V^ 


Appraisal  Activity  Assessment 

CM  Ml  Process  Areas 

-  For  each  process  area  (PA)  a  unique  USATS  stat  code 


^.#1 
■V  if 'iSfi 


was  created  which  allowed  effort  to  be  tracked  at  a  lower 
level  than  just  PHD  work 


-  Each  PHD  PA  contained: 

•  Project  Data  (or) 

•  Task  Data  (or) 

•  Both  Project 
and  Task  Data  (or) 

•  Organizational  Data 


Legend: 


Project  Only 
Task  Only 
^  Project  &  Task 
Organization 
Labor  Hours 


US^ 


Objective  Evidence  &  Time 


^  cf 
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2009  Re-Appraisal  Timeline 


Assumptions:  Based  on  2006  information  &  contract  business  needs 

Normal  SEI  path  (Class  C,  Class  B,  Readiness  Review,  SCAMPI  A) 

Available  work  to  appraise  on  shuttle  work 
Grade  A  mentality--No  risk 
Activities:  Lead  Appraiser  selected 

Appraisal  Plan  drafted  |  | 


Trigger: 

Activities: 


Replan  results: 


3E: 


8  Month  Schedule 


SCAMPI  A 


Need  to  complete  appraisal  activities  in  CY2009 
PHD  format  and  changes  agreed  to 
Focus  Projects  and  tasks  identified 
Appraisal  team  personnel  identified 
Started  Class  C’s  for  PA’s  with  highest  risk 

Gave  indication  of  minimal  gaps  and  drifts 
Lean  Methodology  approach  discussed  (smaller  team,  fewer  appraisal  days 
All^class  C’s  conducted  ASAP  -  reducing  possible  rework 


nth  Schedule 


SCAMPI  A 


Trigger: 

Activities: 


Replan  results: 

US^ 


Contractual  need 

Risks  were  acceptable  with  mitigation 

Discussions  of  business  needs  and  value  of  SCAMPI  B  vs  SCAMPI  A 

Completed  PHD  workshops  and  Class  C's 

DAR  performed  to  assess  possibility  of  schedule  reduction 

DAR  results  provided  feasibility  to  pull  schedule  to  left 
SCAMPI  B  removed 

Grade  A  mentality  change-recognition  of  weaknesses 

IPPD  included  in  scope  of  appraisal 

4th  project  added  as  non-focus  task  for  2  PA’s 

Provided  100%  coverage  across  entire  organization 


Planning 

Appraisal  Activities 
Replan  #1 
Replan  #2 
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PHD  Measures 


•  On  average  the  time  spent  populating  a  PHD 
“cell”  is  approximately  30  minutes/cell 

-  Populating  a  “cell”  means 

•  Interpreting  CMMI  model  and  identifying  type  of  artifact  from 
organization  that  provides  compliance 

•  Providing  Black  Text  artifact  name 

•  Providing  Green  Italic  Text  descriptions 

-  How  the  objective  evidence  meets  the  intent  of  the  CMMI  model 
practice 

•  Providing  associate  link  to  artifact 


USA 


No  matter  how  much  (or  little)  PHD  evidence  you 
need  to  collect  and  populate,  you  can  estimate  the 
effort  needed  to  complete  PHD  work. 
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PHD  example 


/ _  PHD  Format  Benefits 

Organizational  Rows 

Provided  mapping  of  model 
practice  to  organizational 
process  documentation. 


Green  Text 

Provided  explanation  of  how  the  OE 
applies  to  the  model.  Resulted  in  getting 
everyone  up  to  speed  and  appraisal  team 
time  savings  (only  looked  at  applicable 
document  sections) 


USA 
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^  K 


M  Artifact  Checklist  Example 


t  :i-^t 


■'  t^'iM> 


Date  Received 

Requestor 

Brief  Description  of  Artifact 

Project 

Artifact  Folder 
SCAMPL 

Date  Scanned 

Hyperlink 

05/28/2009 

Robin  Hurst 

Integrated  Data  Systems  Configuration  Control  Board 
Operations 

USA004623  Rev  6-Errata 

All 

Docs  All  Projects 

Softcopy 

.ADocs  All  Proiects\USA004623.Ddf 

05/28/2009 

Robin  Hurst 

IPS  System  Software  Technical  Review  Panel 
IDS-SSWA-087  Rev  F  ( 

SysSw 

Docs  All  Projects 

softcopy 

.ADocs  All  Proiects\IDS-SSWA-087.Ddf 

05/28/2009 

Robin  Hurst 

IPS  Application  Software  Technical  Review  Panel 
USA004732  Rev  7  ( 

Appsw/MM 

Docs  All  Projects 

Softcopy 

.ADocs  All  Proiects\USA004732.Ddf 

05/29/2009 

Dreama  Poff 

Verification  &  Validation  Test  Plan 

IDS-VAL-047 

SysSw 

Syssw  Artifacts 

Softcopy 

.  .\S vssw  Artifacts\l  DS-VAL-047 .  odf 

05/29/2009 

Dreama  Poff 

System  Software  Documentation  Standards 

80K61006  Rev  2 

SysSw 

Syssw  Artifacts 

Softcopy 

..\Svssw  Artifacts\80K61006.Ddf 

05/29/2009 

Dreama  Poff 

System  Software  Engineering  Standards 

80K61127 

SysSw 

Syssw  Artifacts 

Softcopy 

..\Svssw  Artifacts\80K61127.Ddf 

Artifact  Checklist  Benefits 


Checklist  Concept 


Provided  Configuration  Management  of  all 
artifacts,  identified  their  requestor,  project 
and  storage  location.  It  also  provided  a 
quick  reference  to  locating  artifact  already 
provided  by  any  person  or  project. 


Hyperlink 


Saved  the  PHD  populators  time  by  being 
able  to  copy  and  paste  the  link  into  the 
PIIDs. 


Allowed  access  to  an  artifact  for  ATMs 
who  didn’t  have  it  in  their  assigned  PA  but 
needed  to  reference  it. 


US^ 
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Noteworthy 
Lead  Appraiser  Traits 


□  Availability  (to  support  you) 

■  Consultations  to  determine  availability 

□  Experience 

■  In  appraising  organizations  with  similar  domains 

□  Soft  Skills 

■  Good  Oral  &  written  communication  skills 

■  Facilitative 

■  Knowledgeable  of  Industry  &  CMMI  Best  Practices 

■  Understanding  cost  effectiveness  and  applicability  to  organization  (not  academic) 

■  Balancing  business  needs  with  compliance 

■  Creative 

■  Effective  leader 


■  May  need  to  alter  the  culture  of  the  organization 

□  Expectations 

■  What  is  expected  from  the  organization 

■  What  is  expected  from  the  LA  -  status  reports,  etc 

□  Resources  (tools,  training  etc) 

■  Available  training  from  LA 

■  Available  consultation  from  LA 
USA'  Tools  LA  requires  for  PHD  or  appraisal  use 
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United  Space  Alliance 


•  I' 


■iU  ■■^^^■:  :■  M 


Advice  to  Others 


>A  .  •  .  'k 

-W» 
f-Ai 


It  can  be  done  faster,  better,  cheaper!! 


JK 


How? 

1.  Maintain  institutionaiization  (Duh!) 

□  Aggressive  PPQA  -  avoid  “drift”  from  process 

□  Active  SEPG  -  evolve/improve  steadily 

2.  Don’t  gold  plate  SCAMPI 

□  Avoid  A+  mentality 

□  External  personnel  (ATM’s  and  LA)  must  be  reasonable 

■  Avoid  unnecessary  rework  from  your  LA 

■  Work  within  existing  PHD  format,  interpretations,  approach 

3.  Be  Lean  and  Green 

□  SCAMPI  Optimization  (fewer  indirects,  scripts,  etc.) 


US^ 


Reduce  PHD  content, 

Reuse  experience  team  members  and 
Recycle  PHD  format  and  scripts. 
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Questions?? 


USA 
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It’s  No  Big  Deal!! 


United  Space  Alliance 


CMMI®  for  Large-Scale,  System  of  Systems  Projects 


9**^  Annual  CMMI  Technology  Conference  and  User  Group 
National  Defense  Industrial  Association  (NDIA) 


Patrick  J.  McCusker 

patrickjmccusker@gmail.com 
November  17,  2009 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


I  Agenda 


►  The  Problem  with  Large-Scale,  System  of  Systems  Projects 

►  Lessons  from  Bridge  Building 

►  How  the  CMMI  can  be  Adapted 

►  CMMI-based  Project  Modeling 


Booz  Allen  Hamilton 


1 


“Make  things  as  simple  as 
possible,  but  not  simpler.  ” 

Albert  Einstein 


“Make  things  as  simple  as 
possible,  but  not  simpler.  ” 

Albert  Einstein 


“For  every  complex  and  difficult 
problem,  there  is  an  answer  that 
is  simple,  easy,  and  wrong.  ” 

H.  L.  Mencken 


When  considering  systems  engineering,  big  is  not  better 

►  There  are  many  examples  of  recent  failures  with  large-scale  projects. 

►  The  Government  Accountability  Office  (GAO)  provides  authoritative  statistics  - 


Examples  of  [Large-Scale]  DOD  Programs  with  Reduced  Buying  Power* 


Program 


InlUal 

estimate 


Joint 

Strike 

Fighter 


Future 

Combat 

System* 


F^22A 

Raptor 


Evolved 

Expendable 

Launch 

Vehicle 


Space 
Based 
Infrared 
System  High 


Expeditionary 

Fighting 

Vehicle 


9199.  S  billion 


SS2.Sbillbn 


$&1.1  billion 


4  billion 


S4.1  bullion 


$3.1  boHion 


Initial  Latest 

quantity  estimate 


ZBGG  aircraft  I20G.  3  billion 


ISaystams  S1275  billion 


aircraft  SG5. 4  billion 


181  vehicles  S28.Cbj|J|Qn 


6  satellites  SI  0.2  billlani 


1 ,025  VBhj  cles  $1 1.1  bj  I  li  0  n 


Latest  Percent  of  unit 
quantity  cost  increase 


2,49Bair:iatl  26.7 


1 5  systems  54  4 


131  aircmft  ies.7 


130  vehicles  137.0 


3  satellites  31 B. 4 


1.025  vehicles  3S.9 


*  GAO,  Assessments  of  Selected  Major  Weapon  Programs,  March  2006,  GAO-06-391 


Booz  Allen  Hamilton 
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When  considering  systems  engineering,  big  is  not  better 

►  There  are  many  examples  of  recent  failures  with  large-scale  projects. 

►  The  Government  Accountability  Office  (GAO)  provides  authoritative  statistics  - 


Examples  of  [Large-Scale]  DOD  Programs  with  Reduced  Buying  Power* 


Program 

InlUal  Initial  Latest  Latest  Percent  of  unit 

estimate  quantity  estimate  quantity  cost  increase 

Joint 

Sinke 

Fighter 

9fg9.Sbllllon  ZBGSaIrcrafi  I20G.3  billion  2,4BB  air:nafl  26.7 

Future 

Combat 

System* 

i 

S&2.S  billion  ISay^stams  S1275  billion  IS  systems  54  4 

- 

Cancelled 

F-22A 

Raptor 

* 

^  . 

$&1 . 1  bll  I  Ion  &4B  aircrafi  SG5 .4  bi  1 II  o  n  131  alrcmft  1  es.  7 

Cancelled 

Evolved 

Expendable 

Launch 

Vehicle 

$15.4  billion  1S1  vehicles  $28.0  l?ilJlon  IseveMcles  137.6 

Space 

Based 

Infrared 
System  High 

1 

$4 . 1  U  1(1 0  n  S  sateN  ites  $1 0 .2  bi  1 II  on  3  satel  iJtes  31 B.  4 

Expeditionary 

Fighting 

Vehicle 

®3.1  hflHion  1.025  vehicles  $11.1bjllion  1.025  vehicles  36.9 

GAO,  Assessments  of  Selected  Major  Weapon  Programs,  March  2006,  GAO-06-391 
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Large-scale  projects  face  common  challenges 


►  The  National  Reconnaissance  Office  (NRO)  found  common  program 
management  flaws  with  large-scale  projects  * 

-  Overzealous  Advocacy 

-  Immature  Technology 

-  Lack  of  Corporate  Roadmaps 

-  Requirements  Instability 

-  Ineffective  Acquisition  Strategy  and  Contractual  Practices 

-  Unrealistic  Program  Baselines 

-  Inadequate  Systems  Engineering 

-  Inexperienced  Workforce  and  High  Turnover 

►  “[Nearly  all  of  the  most  important  and  costly  projects]  continue  to  cost 
significantly  more,  take  longer  to  produce,  and  deliver  less  than  was 
promised.”  ** 


*  Best  Practices  for  Large-Scale  Federal  Acquisition  Programs,  Steven  Meier,  Ph.D.,  PMP,  (National  Reconnaissance  Office) 

**  U.S.  Government  Accountability  Office,  Assessments  of  Selected  Weapon  Programs,  Mar.  2008,  GAO-08-467SP 


The  definition  of  a  “System  of  Systems”  (SoS)  is  stiii  being 
deveioped 


►  A  configuration  of  systems  in  which  component  systems  can  be  added/removed  during  use;  each 
provides  usefui  services  in  its  own  right;  and  each  is  managed  for  those  services.  Yet,  together  they 
exhibit  a  synergistic,  transcendent  capabiiity. 

System -of-Svstems  Engineering  for  Air  Force  Capability  Development.  July  2005,  U.S.  Air  Force  United  States  Air  Force  Scientific  Advisory 
Board 

►  A  set  or  arrangement  of  systems  that  resuits  when  independent  and  usefui  systems  are  integrated  into 
a  iarger  system  that  delivers  unique  capabiiities  [DoD,  2004(1)].  Both  individual  systems  and  SoS 
conform  to  the  accepted  definition  of  a  system  in  that  each  consists  of  parts,  relationships,  and  a  whole 
that  is  greater  than  the  sum  of  the  parts;  however,  although  an  SoS  is  a  system,  not  all  systems  are 
SoS. 

Systems  Engineering  Guide  for  Systems  of  Systems,  Version  1.0  August  2008,  Director,  Systems  and  Software  Engineering,  Deputy  Under 
Secretary  of  Defense  (Acquisition  and  Technology),  Office  of  the  Under  Secretary  of  Defense 

►  A  system  of  systems  is  a  “supersystem”  comprised  of  other  elements  that  themselves  are  independent 
complex  operational  systems  and  interact  among  themselves  to  achieve  a  common  goal.  Each 
element  of  an  SoS  achieves  well-substantiated  goals  even  if  they  are  detached  from  the  rest  of  the 
SoS. 


Mo  Jamshidi,  System  of  Systems  Engineering:  Innoyations  for  the  21^*  Century,  John  Wiley  &  Sons,  Inc.,  Hoboken,  New  Jersey,  2009 


Booz  Allen  Hamilton 


7 


A  DoD  study  of  SoS  provides  useful  insights 


►  Identified  several  current  SoS  programs  - 


Name 

Acronym 

Owner 

Approach 

Army  Battle  Command  System 

ABCS 

Army 

Acquisition  Program 

Air  Operations  Center 

AOC 

Air  Force 

Acquisition  Program 

Ballistic  Missile  Defense  System 

BMDS 

Joint 

Acquisition  Program 

USCG  Command  8i  Control  Convergence 

C2  Convergence 

Coast  Guard 

Strategy 

Common  Aviation  Command  8i  Control  System 

CAC2S 

Marine  Corps 

Acquisition  Program 

Distributed  Common  Ground  Station 

DCGS-AF 

Air  Force 

Program  Office 

DoD  Intelligence  Information  System 

DoDElS 

Intel 

DIA  CIO  Initiative 

Future  Combat  Systems 

FCS 

Army 

Program  Office 

Ground  Combat  Systems 

GC5 

Army 

Program  Executive  Office  PEO 

Military  Satellite  Communications 

MILSATCOM 

Joint 

AF  Wing 

Naval  Integrated  Fire  Control  -  Counter  Air 

NIFC-CA 

Navy 

SE  Integrator  in  PEO 

National  Security  Agency 

NSA 

Intel 

Agency 

Naval  Surface  Warfare  Center  Dahlgren 

NSWC 

Navy 

Warfare  Center 

Single  Integrated  Air  Picture 

SIAP 

Joint 

Acquisition  Program 

Space  and  Missile  Systems  Center 

SMC 

Air  Force 

SE  Authority 

Space  Radar 

SR 

Joint 

Acquisition  Program 

Theater  Joint  Tactical  Networl^ 

TJTN 

Joint 

PEO 

Theater  Medical  Information  Systems  -  Joint 

TMIP 

Joint 

Acquisition  Program 

►  Defined  four  types  of  SoS:  Directed,  Collaborative,  Virtual,  and  Acknowledged. 


Booz  Allen  Hamilton 
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SoS  literature  also  shows  that  like  large-scale  projects,  they  face 
common  challenges  as  well  * 

►  System  elements  operate  independently 

►  System  elements  have  different  life  cycles 

►  The  initial  requirements  are  likely  to  be  ambiguous 

►  Complexity  is  a  major  issue 

►  Management  can  overshadow  engineering 

►  Fuzzy  boundaries  cause  confusion 

►  SoS  engineering  is  never  finished 


*  INCOSE  Systems  Engineering  Handbook,  v  3.1 


There  appears  to  be  some  overlap  in  the  challenge  set  for  these 
two  types  of  projects 


Large-Scale  Project  Challenges  (NRO) 

1.  Overzealous  Advocacy 

2.  Immature  Technology 

3.  Lack  of  Corporate  Roadmaps 

4.  Requirements  Instability 

5.  Ineffective  Acquisition  Strategy  and 
Contractual  Practices 

6.  Unrealistic  Program  Baselines 

7.  Inadequate  Systems  Engineering 

8.  Inexperienced  Workforce  and  High 
Turnover 


SoS  Project  Challenges  (INCOSE) 

1 .  System  elements  operate  independently 

2.  System  elements  have  different  life  cycles 

3.  The  initial  requirements  are  likely  to  be 
ambiguous 

4.  Complexity  is  a  major  issue 

5.  Management  can  overshadow  engineering 

6.  Fuzzy  boundaries  cause  confusion 

7.  SoS  engineering  is  never  finished 


_ 

GAO 

Nep<iri  lo  CoAKressionai  Committees 

DEFENSE 

ACQITSITIONS 

Signinranf  (’Iialicnges 
Ahead  in  Developing 
and  Demonstrating 
Future  Combat 

System's  Network  and 
Software 

AjQ  a  o 

Software  Practices 
Have  Been  Adopted, 
blit  Implementation 
Has  Been  Hampered 
by  Evolving 
Requirements 


Tlip  Anny  and  TST  liavp  afloptpd  a  niimhpr  of  dlscipllnpri 

uractices.  but  their  effective  miplpmentation  at  the  software  developer 

level  has  bppn  haniperpci  by  pvohing  system-level  rpqiiirements.  In 

accordance  with  CMMI"‘  and  under  the  advlsoiy  of  the  Software 
Engineering  Institute,  the  Aiiny  and  LSI  have  adopted  sofl  ware  practices 
that  are  knowTi  to  be  successful  in  fostering  qtiality  software  development, 
such  as  disciplitied  processes,  structiu'ed  management  re\iew  processes, 
and  an  “evolutionary"  development  process.  In  our  analysis  of  five  FCS 
soflware  developers,  we  fotmd  that  requirements  managentent  was  the 
cause  of  most  problents,  indicating  that  a  key  practice  for  managing  and 
developirig  requii  ements  has  not  been  effectiv^ely  implemented  for  the  five 
softw'are  packages  reriewed. 
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“It  is  tradition  in  this  untraditional 
software  field  for  everyone  to  do 
things  his  own  way.  We  are  still 
in  the  prehistoric  age.  ” 

Robert  N.  Britcher 
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Probability  of  Success 


We  know  that  projects  use  technology  and  technology  changes 
over  time... 


Time 
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Probability  of  Success 


Normally  the  progression  of  technical  capabilities  is  predictable 
and  widely  understood... 


Time 
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Probability  of  Success 


But,  technical  advancement  is  not  always  linear,  planned, 
predicted,  controlled,  understood,  or  acknowledged... 


£ 


¥ 


Time 
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Probability  of  Success 


Those  project  managers  that  attempt  to  build  with  new  technology 
bare  the  greatest  risk 


►I 


Time 
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I  Agenda 


►  The  Problem  with  Large-Scale,  System  of  Systems  Projects 

►  Lessons  from  Bridge  Building 

►  How  the  CMMI  can  be  Adapted 

►  CMMI-based  Project  Modeling 
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Perhaps  the  progression  of  bridge  buiiding  through  the  ages  might 
provide  usefui  insights 


17 


New  technical  capabilities  such  as  steel  and  calculus  created 
opportunities  and  threats 
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The  Brooklyn  Bridge  Project  exhibited  many  of  the  challenges  we 
see  with  Large  Scale,  SoS  projects  today 


►  Project  Duration:  14  years 

-  Construction  began:  January  3, 
1870 

-  Opening  date:  May  24,  1883 

►  Length:  5,989  feet 

-  Longest  in  the  world  by  50% 

-  Remained  the  longest  for  20 
years 

►  Cost:  $16,000,000  ($270M  today) 

►  Builders:  John  Roebling,  then 
Washington  &  Emily  Roebling 


Booz  Allen  Hamilton 


19 


The  bridge  was  a  very  dangerous  project,  especially  for  the  project 
manager 
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There  were  several  key  enablers  of  success  for  the  Brooklyn 
Bridge  Project 


►  Project  management 

-  “Owned”  the  design  and 
implementation 

-  Excellent  succession 
planning 

-  Leadership 

►  Technical  leadership 

-  Detailed  designs  developed 
prior  to  construction 

-  Understood  the  risks 

►  Engineering  management 

-  Used  the  best  practices  of 
that  time 

-  Highly  respected 
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I  Agenda 


►  The  Problem  with  Large-Scale,  System  of  Systems  Projects 

►  Lessons  from  Bridge  Building 

►  How  the  CMMI  can  be  Adapted 

►  CMMI-based  Project  Modeling 
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Interface  management,  as  paif  of  Product  Integration  (PI),  becomes 
more  difficult  with  each  added  system 

►  A  critical  aspect  of  product  integration  is  the  management  of  internal  and  external  interfaces 
of  the  products  and  product  components  to  ensure  compatibility  among  the  interfaces. 
Attention  should  be  paid  to  interface  management  throughout  the  project.  * 

►  Large-scale  SoS  projects 
have  difficulty  managing 
interfaces  because  - 

-  Size/scale 

-  Unpredictable 

-  Uncontrollable 

-  Poorly  understood 

►  If  it  is  difficult  to  manage  a 
big  project  when  the 
external  environment  is 
stable,  it  is  infinitely  more 
difficult  to  do  so  when  it  is 
changing. 


*  CMMI  for  Development,  Version  1.2,  (Product  Integration  Process  Area) 
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Modeling  and  Simulation  (M&S)  is  a  primary  method  used  for 
Decision  Analysis  and  Resolution  (DAR)  in  the  overall  SE  process  * 


►  M&S  can  reduce  risk  throughout  the  SE  process,  especially  during  the  early  phases  of 
the  project. 
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High  quality  M&S  becomes  much  more  difficult  when  developing  a 
large-scale,  SoS 


M&S 

Analyses 


Number  of  users, 
topologies, 
availability 


Alternative  analyses, 
interface 
requirements, 
system  performance 


Concept  of 
Operations 


Architecture 

Development 


y  SoS  Engineering 
'  Management 


A 


System  Engineering 
Management  of 
Subordinate  Systems 
Development 


J 
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CMMI  capability  levels  can  be  adapted  to  help  manage  greater 
complexity 


General  Structure  of  the  Capability  Levels  for  each  Process  Area 


Performed 

Managed 

Defined 

Quantitatively 

Managed 

Optimized 

Specific  Practice  1 
Specific  Practice  n 

Generic  Practice  1 
Generic  Practice  n 

Generic  Practice  2 
Generic  Practice  n 

Generic  Practice  3 
Generic  Practice  n 

Generic  Practice  4 
Generic  Practice  n 

Generic  Practice  5 
Generic  Practice  n 

►  Parse  the  capability  levels  - 

-  People:  what  specific  staff  need  to  be  in  place  to  achieve  the  planned  performance? 

-  Process:  what  are  the  specific  process  results  that  will  indicate  success? 

-  Tools:  what  specific  tools  will  be  needed  to  perform  the  process? 

-  Documentation:  what  specific  document  should  be  produced? 

►  Apply  the  capability  level  at  both  the  system  (project)  level  and  SoS  level  (program, 
enterprise) 
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For  each  process  area,  the  capability  levels  can  be  refined  such 
that  organization-specific  metrics  can  be  identified 


1.  Performed 

2.  Managec 

3.  Defined 

4.  Quantitatively 
Managed 

^  Optimized 

People 

Specific  Practice  1 
Specific  Practice  n 

Generic  Practice  1 
Generic  Practice  n 

System 

Generic  Practice  2 
Generic  Practice  n 

System 

Generic  Practice  3 
Generic  Practice  n 

System 

Generic  Practice  4 
Generic  Practice  n 

System  i 

/Generic  Practice  5  " 
Generic  Practice  n 

System 

SoS 

SoS 

SoS 

SoS 

/  SoS 

Process 

Specific  Practice  1 
Specific  Practice  n 

Generic  Practice  1 
Generic  Practice  n 

System 

Generic  Practice  2 
Generic  Practice  n 

System 

Generic  Practice  3 
Generic  Practice  n 

System 

Generic  Practice  4 
Generic  Practice  n 

System 

Gener^Practice  5 
Genej/c  Practice  n 

System 

SoS 

SoS 

SoS 

SoS 

SoS 

Tools 

Specific  Practice  1 
Specific  Practice  n 

Generic  Practice  1 

Gpswfic  PTJL+Hien 

System 

Generic  Practice  2 
Generic  Practice  n 

System 

Generic  Practice  3 
Generic  Practice  n 

System 

Generic  Practice  4 
Generic  Practice  n 

System 

/enenc  Practice  5 
generic  Practice  n 

System 

SoS 

SoS 

SoS 

Sos/ 

SoS 

Documentation  . 

Specific  Practiced 

Specific  Practice  n 

Generic  Prq^ce  1 
^ieneric 

VSystem 

Generic  Practice  2 
Generic  Practice  n 

System 

Generic  Practice  3 
Generic  Practice  n 

System 

Generic  Practice  4 
Generic  Practice  n 

Sy^m 

Generic  Practice  5 
Generic  Practice  n 

System 

/  SoS 

SoS 

SoS 

/soS 

SoS 

Example  1  -  Product  Integration,  Level  1, 

Example  2  -  Product  Integration,  Level  5,  Peopie 

Documentation  Requirements 

Requirements 

►  Integration  Plan 

►  PI  staff  understand  and  contribute  to  process 

optimization  activities 

►  Integration  Procedures 

►  Appropriately  skilled  and  trained  staff  are  assigned 

►  Integration  Criteria 

to  monitor  PI,  support  root  cause  analyses,  and 
implement  PI  process  improvements. 

Booz  Allen  Hamilton 


27 


Product  Integration  (PI)  processes  might  be  more  quickly  assessed 
and  problem  areas  targeted  for  improvement 


► 


PI  challenges  with  large- 
scale  SoS  projects  - 


Architecture 

Development 


-  Disconnect  between 
subordinate  projects 

-  Disconnect  between 
subordinate  projects  and 
the  SoS  program 

-  Disjoint  business 
practices 

-  Diverse  vendor  or 
integrator  contract 
requirements 


SoS  Engineering 
Management 


J 

\ 


System  Engineering 
Management  of 
Subordinate  Systems 
Development 


J 
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I  Agenda 


►  The  Problem  with  Large-Scale,  System  of  Systems  Projects 

►  Lessons  from  Bridge  Building 

►  How  the  CMMI  can  be  Adapted 

►  CMMI-based  Project  Modeling 


Booz  Allen  Hamilton 


29 


Business  Process  Management  (BPM)  technology  might  be  used  to 
better  plan  and  manage  large-scale,  SoS  projects 


►  Common  BPM  capabilities  allow  for  - 

-  Modeling  a  process,  typically  in  a 
graphical  format 

-  Integrating  a  variety  of  processes, 
external  applications,  and  databases 
with  the  defined  process 

-  Managing  step-by-step  process 
execution  across  multiple  personnel 
roles 

-  Creating  exception  handling  and 
alternative  processes 

-  Monitoring  the  health  and  fulfillment 
cycle  of  the  process 

-  Assigning  roles  to  personnel  either  by 
user  direction  within  the  process  or 
based  on  current  workload  queues 

-  Collecting  metrics  on  process 
execution 

-  Simulating  the  execution  of  the  defined 
process  based  on  either  empirical 
results  or  user-provided  parameters 


TileNet' 


N  T  A  L  I  O 


METASTORM 


Thinkiiquidr 


APPIAN 


IBTIBCO 


Adobe 
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As  an  example,  we  can  use  the  PI  process 


Context  Diagram  for  the  Integration  Process  * 


Controls 

-  Agreements 

Project  procedures  &  processes 


Inputs 

-  Definition  of  system 
hierarchy 

-  Architectural 
design  requirements 

-  Supplied  system  elements 

-  Integration  Plan 

-  Id entif  cation  of  external 
systems 

-RV™ 


Activities 

-  Define  integration  strategy 
Schedule  system  elements  and  enabling 
systems  per  planned  deliveries 
-Integrate  system  elements 
-Record  integration  information 


Enablers 

-  Enterprise  Infrastructure 
-  Enterprise  Policies,  Processes  &  Standards 
-  Integration  enabling  systems 


Outputs 

-  Verifiable  system 

-  Results  of  integration  testing 

-  Problem  resolution  records 

-  Interface  Control 
Documents  (I CDs) 

-  Updated  RVTM 


INCOSE  Systems  Engineering  Handbook,  v.3.1 
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Since  integration  processes  must  occur  at  each  ievei  of  the  SoS 
hierarchy,  they  can  be  modeied  to  support  project  pianning 


►  Level  of  Effort  (LOE) 

►  Documentation 

►  Review  cycles 

►  Staffing  requirements  to 
Quantitatively  Manager 
and/or  Optimize 

►  Tool  and  database 
requirements 


I  I  I 


►  Organizational  issues  and 
communications  flow 


In  summary... 


►  Large-Scale,  SoS  projects  are  challenged  on  many  fronts. 

►  Project  Managers  are  not  equipped  to  make  excellent  decisions. 

►  One  key  issue  is  that  standard  processes  tend  to  break  down. 

►  Large-Scale,  SoS  projects  are  much  more  complicated  and  therefore  the 
planning  (i.e.,  project  modeling)  and  management  (i.e.,  monitoring, 
assessments,  control,  improvement)  of  engineering  processes  must  also  be 
more  sophisticated. 

►  The  CMMI  community  can  help  with  this  problem  by  adapting  proven 
methodologies  so  that  they  can  be  readily  applied  to  these  larger  projects. 
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I  am  happy  to  take  your  questions  and  look  forward  to  hearing 
your  thoughts!  _ _ _ 


Booz  Allen  Hamilton 


34 


Making  CMMI  Level  5  Statistical 
Principles  Palatable  to  an 
Employee-Wide  Demographic 


<OST 

CMMI  LEVEL  3  |  ISO  9001:2008 


Agenda 


<  Couple  of  questions 

<  Session  1 

<  What  worked 

<  Additional  Resources 


<OST 


Lets  get  a  current  baseline  of  CNN  viewers... 


<OST 


Do  you  watch  CNN  everyday? 


Anderson  Cooper. . . 


<OST 


Do  you  know  who  Anderson  Cooper  is? 
http://video.qooqle.com/videosearch?sourceid= 

navclient&rlz=lT4RNWN  enUS284US284&q=an 

derson%20cooper%20cnn%20heroes&hl=en&u 

m=l&ie=UTF-8&sa=N&tab=wv# 


Role  Play 
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<  Approach  for  this  segment 


Lets  establish  a  baseline  of  current  understanding...  <OST 


Variable  “A”  follows  a  lognormal  distribution  as 
determined  by  the  A-D  goodness  of  fit  test.  The 
Chi-Square  test  shows  it  has  a  p-value  of  0.002. 


Roll  a  dice... 


t 


■D 

O 


0 


2  3 


Values  ^ 
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Coin  toss... 


t 


■D 

O 

O 


0 


1/2  - 


Head 


Tail 


Values  ^ 
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Coin  toss... (unfair  coin) 


2/3  - 


t 


■D 

O 


0 


Head 


Tail 


Values  ^ 
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Two  die... 


0.20 


0.15 

i  0.10 

o 

I— 

CL 

0.05 

0.00 


Probability... 0  to  1;  0%  to  100% 


Die  Roll 
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■  1 
■  2 

■  3 

■  4 

■  5 

■  6 


Area  under  a  curve... 
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> 


100% 


Heights  of  the  VRVL  PAT. . .+  QA  group+  ESEPG  <OST 


Joann  -  5'  1" 
Lakshmi  -  5'  4" 
Domenic  -  5'  8" 
Krithika  -  5'  8" 
Hongda  -  6'  1" 
Jerome  -  6'  1" 
James  -  6'  3" 
Charlotte  -  5’6" 
Carolina  -  5'5" 
Surya  -  5'9" 
Nalini  -  5'0" 
Barbara  -  5'5" 
Nishi  -  5'2" 
Deepti  -  5'6" 
Dan  Renfroe  - 
Joe-5'11 
Jack-5'10" 

Ajay  -  6'0" 

Vijay  -  5'5" 


Histograms.. .PDF 
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Histogram  of  VRVL  PAT^  VRVL  +  QA  group 
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Histogram  of  VRVL  PATj,  VRVL  +  QA  group^  VRVL  +  QA  group  + 
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Heights  in  general... 


Number  of 
people 


5‘8" 


Height 
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Some  other  distributions... 


Jjamnalj  Triangular  Unifomn 


Min  Extreme  Logistic 


Student's  t 


Lognomnal 


Geometric 


Discrete  Uniform 


L  II  IJIIiii 


Yes-No 


Custom 
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Anderson  -  Darling... Does  this  fit? 


Name  Is  a  good  fit  if.. 


Anderson-Darling  A-D<1.5 

Chi-Square  PValue>0.05 

Kolmogorov-Smirnov  K-S<0.03 
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0  Anderson-Darling 
0  Kolnogorov-Snirnov 
OChi-Sguare 


More  details 


Oldest 


Results  of  Goodness  of  fit  test. .  .for  heights 
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Do  we  understand... 


A  distribution... Goodness  of  fit 
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..p-value 


Lets  get  another  baseline... 
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<  Variable  “A”  follows  a  lognormal  distribution  as 
determined  by  the  A-D  goodness  of  fit  test.  The 
Chi-Square  test  shows  it  has  a  p-value  of  0.002. 


And  CNN  viewership?? 


<  What  is  it  now? 
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Which  baselines  changed? 
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<  CNN  Viewership's? 

<  Do  you  know  what  Anderson  Cooper  looks  like? 

<  Your  “assumed  identities”  understanding  of  the 
statement? 


Why/Why  not? 


other  tests 
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<  chi-square  -  The  oldest  and  most  common  goodness-of-fit  test.  This  test  gauges 
the  general  accuracy  by  breaking  down  the  distribution  into  areas  of  equal 
probability  and  comparing  the  data  points  within  each  area  to  the  number  of 
expected  data  points.  Generally,  a  p-value  greater  than  0.5  indicates  a  close  fit. 

<  Kolmogorov-Smirnov  -  A  goodness-of-fit  test,  the  result  of  which  is  essentiaily  the 
largest  vertical  distance  between  the  two  cumuiative  distributions.  Generaliy  a 
value  less  than  0.03  indicates  a  good  fit. 

<  Anderson-Darling  -  A  goodness-of-fit  test  that  closely  resembles  the  Kolmogorov- 
Smirnov  test,  except  that  it  weights  the  differences  between  the  two  distributions  at 
their  tails  greater  than  at  their  mid-ranges.  Use  this  test  when  you  need  a  better  fit 
at  the  extreme  tails  of  the  distributions.  Generally  a  value  less  than  1 .5  indicates  a 
close  fit. 


Role  play  over 
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<  Thank  you  for  the  role  play 


stuff  that  worked  or  we’d  do  differently 
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4  Worked  well 

►  Socratic  method 

►  Their  data 

►  Everyday  examples 

►  Everyone  participates 

►  No  fear 

<  What  we  are  doing  differently 

►  More  exercises 

►  Come  up  with  x  and  y  factors  sooner 


What  we  did  next... Assignment 
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Assignment  - 

Y  Factor  -  Identify  a  problem/somethingyou  would  like  to  be  able  to  predict 

•  Related  to  work 

•  Having  a  few  options  is  ideal  as  opposed  to  just  one  issue 

X  -  Factor  -  For  each  problem  identify  the  possible  "x  factors"  that  have  an  impact  on  the  problem 

•  Some  factors  may  be  underyour  control  (label  them  as  controllable) 

•  Some  may  be  outside  your  control  (label  them  as  uncontrollable) 

•  tf  you  are  not  sure  whether  or  not  it  makes  an  impact  (you  think  it  may)^  put  it  in  any  ways... more  the  merrier  right  now 


Meeting  -  After  completing  the  above  two  steps,  set  up  a  meeting  with  Deepti 


Results 
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<  Sample  models  that  came  up 

►  Time  I  need  to  spend  on  the  help  desk 

►  Time  to  develop  reports 

►  Quality  of  end  product 

►  My  productivity 

►  LOE  needed  for  Testing 

►  Etc. 


What’s  in  the  Appendix? 


<OST 


<  Material  from  additional  sessions 

<  If  you  need  help,  we’ll  be  happy  to  chat  via: 

►  Email 

►  Phone 

►  Etc. 


...to  provide  tips  from  our  experience 


Our  Contact  Information 


Nishi  Narula 

1  Director,  Business  Performance 


WWW.OST6LOBAL.COM 


◄osr 

CMMI  lEVElsltSO  9001:2000 


2001 M  Street,  NW,  Suite 
Washington,  DC  20036 

T  202.467.7650 
F  202.466.8117 

nnarula@ostglobal.com 
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Questions 
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Appendix 
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Some  more  samples 
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<  Hypothesis  Tests  and  Jokes 

<  Standard  Deviation  and  Variability 

<  Two  real  life  models 

<  L5  and  basic  steps 
^Tool  trainings 


Do  you  understand  this  statement... 
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<  The  NULL  hypothesis  was  rejected  in  favor  of  the 
alternative  hypothesis  since  the  p  value  was... 

<  Let’s  get  a  baseline  from  the  class 

►  How  many  Get  It? 

►  Somewhat? 

►  Do  NOT  get  it? 


■  ■  ■ 
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Have  you  heard 


◄  Innocent  until  proven  guilty 


The  person  is  innocent 


The  person  is  guilty 


Null  Hypothesis 


Alternative 

Hypothesis 


Which  of  the  above  statements  is... 

◄  Status  Quo 

◄  Conventional  wisdom 

◄  Doesn’t  need  to  be  proved 

◄  Accepted  without  additional  proof 


Null  Hypothesis 


Alternative  Hypothesis 
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<  Not  Status  Quo 

<  Not  Conventional  wisdom 

<  The  burden  of  proof  rests  on 

•  One  who  challenges 

•  Makes  a  new  claim 

•  Wants  to  change  the  status  quo 


So  what’s  a  hypothesis  test? 


...To  Suppose 

A  pair  of  statements  (not 
questions) 

Can  be  tested 

Has  a  clear  yes/no  answer 

If  one  Is  true  the  other  is  false 

Nothing  “slips  through  the  cracks” 


Ho:  Dan’s  height  is  5.11. 
Ha:  Dan’s  height  is  6.2. 


Ho:  Is  Surya  a  Doctor? 
Ha: 
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Ho:  Djindo  is  on  the  AITS  project. 

Ha:  Djindo  is  not  on  the  AITS  project. 


Ho:  Sonu  is  on  the  AITS  project. 
Ha:  Jadrana  is  not  on  the  HSEEP 
project. 


Ho:  Is  Arthur  really  tall? 
Ha: 


So  what’s  a  hypothesis  test? 
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^  ...To  Suppose 

^  A  pair  of  statements  (not 
questions) 


Ho:  Kusum  was  born  in  Washington  DC. 

Ha:  Kusum  was  not  born  in  Washington  DC. 


Can  be  tested 
Has  a  clear  yes/no  answer 
If  one  Is  true  the  other  is  false 
Nothing  “slips  through  the  cracks” 


Ho:  Julie  is  a  doctor. 

Ha:  Julie  is  not  a  doctor. 


Ho:  Sean  watches  CNN  every  day. 
Ha:  Sean  does  not  watch  CNN  every 
day. 


So  what’s  a  hypothesis  test? 
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^  ...To  Suppose 

A  pair  of  statements  (not 
questions) 


Ho:  Apple’s  help  desk  answers  calls  in  2  min 
or  less. 

Ha:  Apple’s  help  desk  answers  calls  in  more 
than  2  min. 


Can  be  tested 
Has  a  clear  yes/no  answer 
If  one  is  true  the  other  is  false 
Nothing  “slips  through  the  cracks” 


Ho:  The  average  GPA  of  GW  is  2.6  or 
higher. 

Ha:  The  average  GPA  of  GW  is  less 
than  2.6. 


Ho:  Dan’s  height  is  5.11  or  taller. 
Ha:  Dan’s  height  is  less  than  5.11. 


Once  you  run  the  hypothesis  test  you  get  a 
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<  P-Value... 


When  the  p  is  low,  the  null  must  gi 

When  the  P  is  high,  the  null  must  fly 


/ 


HIGH 


Less  than  (<)  0.05  or  5% 


Equal  to  or  Greater  than  (<=)  0.05  or  5% 


So  what’s  a  hypothesis  test? 

<0ST 

Mpple’s  help  desk  answers  calls  in  2  min  !  1 
1  or  less.  1 

1  Ha:  Apple’s  help  desk  answers  calls  in  more  1 
1  than  2  min.  1 

Ho:  The  average  GPA  of  GW  is  2.6  or 
higher. 

Ha:  The  average  GPA  of  GW  is  less  . 
than  2.6.  -- — 


0.0^ 


Ho:  Dan’s  height  is  5.11  or  taller. 
Ha:  Dan’s  height  is  less  than  5.11 


Do  you  understand  this  statement... 
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<  The  NULL  hypothesis  was  rejected  in  favor  of  the 
alternative  hypothesis  since  the  p  value  was... 

<  Let’s  get  a  baseline  from  the  class 

►  How  many  Get  It? 

►  Somewhat? 

►  Do  NOT  get  it? 


Technical  Terms... P  value  Is  the 


<OST 


<  Probability  of  Null  being  true... or  accepting  the  null 


95.4% 


99.7% 


Questions 


<OST 


Some  terms... 


< CENTRAL TENDENCY 

►  Mean/Average 

►  Median 


◄  VARIANCE 


<OST 


The  weakness  of  using  Central  tendency  alone... 
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May  be  a  far  cry  from  reality. . . 


The  weakness  of  using  Central  tendency  alone... 
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You  are  a  Manager  for  Boeing. 

Expectation  -  50  jet  engines  on  June  29^^. 

You  will  store  them  in  a  warehouse. 

Each  day  (late)  -You  loose  $2  million. 

Each  day  (early)  -  You  pay  $50,000  extra 
for  warehouse  costs. 

Bidder  A:  Average  20  days 


Bidder  B:  Average  20  days 


Past  deliveries... 


<OST 


<  Bidder  A 

►  1 0  days 

►  20  days 

►  30  days 

<  Bidder  B 

►  1 9  days 

►  20  days 

►  21  days 


•  Bidder  A 
-10,0,10 


•  Bidder  B 
-1,0,1 


Not  a  good  measure  of  reliability... 


So  how  do  we  measure  it? 
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<-10, 0,10 

<  Square  them  first  - 1 00, 0, 1 00 

<  Divide  by  total  number  of  values 

<  Square  root 

<  Standard  Deviation  - 


cr  = 


Some  more  terms... 


<  baselines 
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What  we  do  is  important 
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Keeping  planes  in  the  air 

<  Ensuring  people  have  access  to  insurance  claims 

<  Ensuring  government’s  money  is  spent  well 

<  Housing  for  the  needy 

<  Homeland  Security 

<  Tracking  progress  for  government  spending 


So  we  need  to  do  well 


<  Continuously... 

<  In  spite  of  the  constraints.. 


<OST 


Variability... 


<OST 


OST 


Corp  Units 


[f 


Project  1  ■  Project  2 


t 


Member  iBMember  ZBMember  3 


Constraint 

1 

Constraint 

2 


Choice  1 


Project  3 


Choice  2 


So  now  that  we  have  variability... what  can  we  do? 
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Blind  Educated  Systematic 

Guessing  Guessing  Prediction 


Make  the  variability  go  away^ 


Great  Concept,  but  does  this  really  work? 
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4  The  Heart  Break  Model 

►  Heart  Break... 

►  Stress  cardiomyopathy 

►  Apical  ballooning  syndrome 

►  Mr.  Li  goes  to  Wallstreet 

►  Default  Correlation 

►  Examples - 

•  Chance  of  a  dairy  farm  going  bust  - 1 0% 

•  Chance  of  a  dairy  going  bust  -  5% 

•  What  if  the  dairy  farm  goes  bust 

•  And  what  if  the  dairy  gets  its  supplies  from  this  dairy... 

•  Chance  of  dairy  going  bust... Rise 


Default  Correlation  Examples... 
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The  love  calculator... 


<OST 


<  Now  that  I  have  a  model... 

<  Don’t  need  conventional  wisdom... 

<  Rating  Agencies... 

<  Lot’s  of  buyers... 

<  Market  explosion... 

►  From  10’s  of  billions  of  $  in  2000  to  $2  trillion  in  2007 

<  Supply  and  demand... 

<  Loans  become  cheap... qualify  easily 
Defaults... 


The  effects... 


<OST 


<  Banks  are  scared  to  lend 

<  Liquidity  dries  up 

<  Businesses  can’t  get  loans 

<  Economy  grinds  to  a  halt... 


What  went  wrong? 
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<  Companies  and  people  are  different 

<  Models  weren’t  updated 

<  Assumptions... 

<  Not  used  as  intended . . . 

<  Understanding... 


Sample  2 


<OST 


<  OST  Accounts 

►  Know  before  you  go  -  Risk  Free! 

•  What  if  scenarios  -  Virtually 

•  Efficiencies 

•  Bottlenecks 

►  Great  Idea?  Synergy! 

►  Accurately  Evaluate  Opportunities 

►  Powerful  clear  communications 

►  Predict  the  future!! 


Process-Oriented  Process-Agnostic 


Ok,  how  do  we  get  there? 
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y 


y 

X 


X 


usiness  Goals 
orecard) 

siness  Goals 
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recesses 
to  a  Vital  x 
sub-process  to  be  tackled 
by  DMAIC  team) 


CreUeitMlee^X 


60 


Jokes 
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MUST  READ 

Three  men  are  in  a  hot-air  balloon.  Soon,  they  find  themselves  lost  in  a  canyon  somewhere.  One  of  the  three  men  says, 

"I've  got  an  idea.  We  can  call  for  help  in  this  canyon  and  the  echo  will  carry  our  voices  far."  So  he  leans  over  the  basket  and 
yells  out,  "Helllloooooo!  Where  are  we?"  (They  hear  the  echo  several  times.) 

Fifteen  minutes  pass.  Then  they  hear  this  echoing  voice: "  Helllloooooo!  You're  lost!!"  One  of  the  men  says,  "That  must  have 
been  a  statistician."  Puzzled,  one  of  the  other  men  asks,  "Why  do  you  say  that?"  The  reply:  "For  three  reasons.  (1)  he  took  a 
long  time  to  answer,  (2)  he  was  absolutely  correct,  and  (3)  his  answer  was  absolutely  useless." 

I  asked  a  statistician  for  her  phone  number...  and  she  gave  me  an  estimate. 

ARGUING  WITH  A  STATISTICIAN  IS  A  LOT  LIKE  WRESTLING  WITH  A  PIG.  AFTER  A  FEW  HOURS  YOU  BEGIN  TO 
REALIZE  THE  PIG  LIKES  IT. 


Then  there's  the  one  that  if  you  laid  every  statistician  on  the  face  of  the  earth  end  to  end  you  wouldn’t  reach  a 
conclusion . Probably. 

There  was  this  statistics  student  who,  when  driving  his  car,  would  always  accelerate  hard  before  coming  to  any  junction, 
whizz  straight  over  it ,  then  slow  down  again  once  he'd  got  over  it.  One  day,  he  took  a  passenger,  who  was  understandably 
unnerved  by  his  driving  style,  and  asked  him  why  he  went  so  fast  over  junctions.  The  statistics  student  replied,  "Well, 
statistically  speaking,  you  are  far  more  likely  to  have  an  accident  at  a  junction,  so  I  just  make  sure  that  I  spend  less  time 
there.” 


Jokes 
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JUST  FOR  FUN 


Three  professors  (a  physicist,  a  chemist,  and  a  statistician)  are  called  in  to  see  their  dean.  Just  as  they  arrive  the  dean  is  called  out  of  his  office, 
leaving  the  three  professors  there.  The  professors  see  with  alarm  that  there  is  a  fire  in  the  wastebasket. 

The  physicist  says,  "I  know  what  to  do!  We  must  cool  down  the  materials  until  their  temperature  is  lower  than  the  ignition  temperature  and  then 
the  fire  will  go  out." 

The  chemist  says,  "No!  No!  I  know  what  to  do!  We  must  cut  off  the  supply  of  oxygen  so  that  the  fire  will  go  out  due  to  lack  of  one  of  the 
reactants." 

While  the  physicist  and  chemist  debate  what  course  to  take,  they  both  are  alarmed  to  see  the  statistician  running  around  the  room  starting  other 
fires.  They  both  scream,  "What  are  you  doing?" 

To  which  the  statistician  replies,  "Trying  to  get  an  adequate  sample  size." 


The  Fake  Software  project 
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Hi  Team - 

Your  our  VRVL  PAT  meeting tomorro\v,  \ve  vM  pretend  to  be  working  on  a  softLvare  project's  proposal. 

We  are  working  on  cost  proposal  v^^hich  needs  estimation.  We  need  to  develop  an  application  for  a  college  that  allows  - 

•  A  alum  to  log  in 

•  Search  a  database  of  alum's  by  last  name,  first  name^  year  of  graduation 

•  Record  up  to  10  personal  contacts 

This  is  the  only  information  we  have  and  we  need  to  respond  to  the  proposal, 

the  window  for  asking  questions  is  over... ha,  ha,  ha!  So  Just  do  your  best  with  the  information  at  hand. 

Here  are  the  responsibilities 


Phase 

Owner/You  are  assigned... 

Requirements 

Dorn 

Design 

Hongda 

Development 

Lakshmi 

Integration 

James 

Test 

Rhiya,  Joann 

UAT 

Jerome,  Sujani 

Please  provide  estimates  for  each  of  the  phases  you  are  assigned  to  in  the  followingformat - 


Category 

Estimate  of  hours 

•  Most  likely  number  of  hours  needed  to 
complete  the  task 

• 

•  Minimum  hours/Best  Case 

• 

•  Maximum  hours/Worst  case 

• 

//  this  whole  exercise,  including  the  optional  step  is  taking  you  mote  than  10  minutes,  please  stop!!  We  will  just  do  it  in  the  meeting  then.  Else,  please  tiy  to  send  it  to  me  any  time  before  the  PAT  meeting. 

One  last  thing,  you  may  want  to  do  a  ^'bottom  up  estimate'".  (Optional  Step) 

•  That  is  jot  down  sub-tasks  E.g.  for  requirements  -  meeting  with  customer,  developing  first  draft,  conducting  peer  review,  fixing  issues,  sign  off  meeting  etc. 

•  Assign  hours  to  each  task 

•  Provide  the  total  in  the  table  above,  but  bringthe  details  to  the  meeting. 
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Background 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  An  organization  adopting  the  CMMI  modei  has  to 
make  numerous  decisions: 

-  Scope  of  the  improvement  effort 

-  Model  representation 

-  I PPD  extension 

-  Structure  of  the  policies  and  processes 

-  Training  program 

-  Measurement  repository 

-  Etc. 

•  These  choices  made  have  a  profound  effect  on  the 
value  of  the  improvements,  the  buy-in  of  the 
organization,  and  the  ultimate  success  of  the  CMMI 
effort 

•  This  tutorial  will  discuss  the  key  decisions  to  be 
2  made  and  options  to  be  considered 


Topics 


MORTHROR  CRUMMJXN 


Defining  the  future 


•  How  decisions  drive  success 

•  Scope  decisions 

-  Organizational  scope 

-  Model  scope 

•  I  nfrastructure  decisions 

-  Policies,  processes,  procedures,  and  plans 

-  Process  asset  library 

-  Measures  and  measurement  repository 

-  Training 


How  Decisions  Drive  Success 


MORTHROP  CRUMMJXN 


Defining  the  future 


Value  of  the  improvements 

Perceived  value  of  the 
improvements 

Success  of  the  improvements 
Cost  of  the  improvements 
Speed  of  the  improvement 
"Dead-ends" 

Fit  with  culture 
Strengthening  of  culture 
Perceived  bureaucracy 
Buy-in 

Ability  to  address  other 
improvement  goals 


Why  Does  an  Organization  Adopt  CMMI? 
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Defining  the  future 


•  CMMI  supports  successful,  predictable  program 
performance 

-  Lowered  cost,  reduced  risk 

-  Industry  data  indicates  Level  3  is  -20%  cheaper  than 
Level  1 

•  CMMI  can  be  a  program  requirement 

-  RFPs  may  call  out  a  requirement  to  be  CMMI  Level  3, 
across  the  team 

-  Primes  are  anxious  to  team  with  CMMI  Level  3  suppliers 

•  CMMi  can  be  a  competitive  discriminator 

-  Demonstrates  your  capabilities,  against  an  well-known 
industry  standard 


What  Does  It  Mean  to  "Adopt"  CMMI? 
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Defining  the  future 


•  Organizations  adopt  CMMI  to  ensure  they  are  implementing 
industry  best  practices 

•  This  requires  appraising  whether  or  not  the  organization  and 
its  projects  are  currently  performing  these  practices 


*  Based  on  the  results  of  an  initial  ("gap") 
appraisal,  the  organization  and  projects 
implement  improvements 


-  Often  requires  new  practices,  clearer 
documentation,  consistency  in  following 
plans  and  processes,  checks  and  balances 

When  the  requires  improvements 
have  been  made,  the  organization 
conducts  a  formal  appraisal  and 
receives  their  Level 
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A  Process  Paradigm 


MORTHROR  GRUMMAN 


Defining  the  future 


Techno^Oigy  R&D  Organisations  Everything  is  a  Process! 
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Jeanine  Siviy  and  Eileen  Forrester,  Accelerating  CM  Ml  Adoption  Using  Six  Sigma,  CM  Mi  Users  Group,  2004 


Adopting  the  CMMI 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  Key  enablers 

-  Willingness  to  learn  unfamiliar  practices 

-  Desire  to  extract  value  rather  than  "check  the  box" 

-  Ability  to  interpret  the  CMMI  in  your  context 

-  Access  to  experts 
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Exercise  -  What  are  your  organization's 
improvement  goals? 


MORTHROR  CRUMMJXN 


Defining  the  future 


•  What  are  your  organization's  business  goais  (beyond 
achieving  some  CMMI  ievel)? 

-  E.g.,  reduce  cost,  increase  quality,  decrease  schedule, 
increase  market  share,  etc. 

•  What  does  senior  management  reaiiy  care  about? 

•  I  n  making  the  changes,  what  shouid  not  change? 


Topics 


MORTHROR  CRUMMJXN 


Defining  the  future 


•  How  decisions  drive  success 

•  Scope  decisions 

-  Organizational  scope 

-  Model  scope _ 

•  I  nfrastructure  decisions 

-  Policies,  processes,  and  procedures 

-  Process  asset  library 

-  Measures  and  measurement  repository 

-  Training 
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Organizational  Scope 
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Defining  the  future 


•  Must  decide  where  to  adopt  the  model 

-  Discipline:  software,  systems,  hardware,  services 

-  Organizational  scope:  project,  business  unit,  division, 
sector,  company 

-  Piloting  vs.  organizational-wide  deployment 

•  Key  considerations 

-  Do  you  know  how  big  the  gaps  are? 

-  How  much  money  and  staff  are  available  to  assist  the 
projects? 

-  Where  can  you  gain  some  early  successes? 

-  Where  are  you  experiencing  the  most  pain? 

-  How  much  resistance  will  there  be  to  the  improvements? 


11 


Exercise 
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Defining  the  future 


•  What  choices  should  ( has)  your  organization  make 
( made)  about  CMMI  adoption? 

-  Organizational  scope 

-  Model  scope 

•  What  information  is  needed  to  make  the  choices  (or 
ensure  the  choices  were  correct)? 
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Topics 


MORTHROR  CRUMMJXN 


Defining  the  future 


•  How  decisions  drive  success 

•  Scope  decisions 

-  Organizational  scope 

-  Model  scope 

•  I  nfrastructure  decisions 

-  Policies,  processes,  and  procedures 

-  Process  asset  library 

-  Measures  and  measurement  repository 

-  Training 
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Project  Use  of  Organizational  Process 
Assets 


MORTHROR  GRUMMAN 


Defining  the  future 


Industry/Government 
Standards 


Organizational 
Policies  &  Processes 


Process  Asset 
Library 


Organization 

Project 


Tailoring 


Project 

examples 


Project  Defined  Process,  Project  Plans, 

Procedures,  &  Standards  Schedules,  &  Budgets 
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Measurement  Organizational 

Repository  Training 


r 

r 


Historical  data 


Project 

Results 


Project-Specific 

Training 


A  Top-Level  Comparison 
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Defining  the  future 


Policy 


High-level  ''what"to  do 
(organizational  guidance) 


Process 


High-level  "how"  to  do 

(organizational  standard,  tailored  by  projects) 


Procedure  Low-level  "how"  to  do 

(details  needed  to  follow  a  strategy) 


I  nstantiation  of  the  process 
(how  often,  when,  etc.) 
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(Organizational)  Policies 


MORTHROP  CRUMMJXN 


Defining  the  future 


"A  guiding  principle  typically  established  by  senior  management  that 
is  adopted  by  an  organization  to  influence  and  determine  decisions." 

-  Glossary,  CMMI-DEV  vl.2 


•  Policies  provide  guidance,  to  Project  Managers  and  other 
functional  groups,  on  required  activities 

(what  to  do) 

•  Example: 

-  "All  projects  shall  establish  and  maintain  a  Risk  Management  Plan" 

•  Performers  follow  their  plans,  processes,  and  procedures,  which 
must  reflect  the  policies 

-  Need  not  be  familiar  with  the  policies 
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Using  Policies 


MORTHROP  CRUMMJXN 


Defining  the  future 


GP  2.1  Establish  an  Organizational  Policy 

Establish  and  maintain  an  organizational  policy  for  planning  and 
performing  the  process. 

•  "Establish  and  maintain"  includes  usage  (see  Glossary)^ 
suggests  someone  must  audit  for  compliance  with  policies 

-  Both  projects  and  functional  groups 
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Constructing  Policies  -  Option  1 


MORTHROP  CRUMMJXN 


Defining  the  future 


’■■mii'ij; 

AJlhough  Ihe  prinar/  EnphasiE.  of  Ihs  Risk.  ManagEnsnt  procBSE  arBa 
is  on  (hiE  pr-ojecL  ihe  ocncepts  can  alsc  be  app  ed  to  manage 
crpanizat onal  rsks. 


Related  Procga-a  Areas 


.RE.feriioi  ihe  PianningpriDceM  anaa  forwi:^&  }nformarKX\  atetri 

ideniilf\Gat<on  Qfp«jeci-  fisks  and  pfann^  Jbr  jivoSemenr  of  rei'evani 
arafce^rciyens. 

ReJer  io  ihe  F^^^iecr  h'fani.itar^  ano'  Owrirai  process  an&a  for  more 
ih/ainahcin  aiour  JTiom'ronng  projeci  risks. 

ReJer  io  ihe  Dscisktn  Analysis  ano'  Reaoiufi'on  prooess  area  Asr  mere 
infainah'on  aiourijiamg  a  .formal' evaJoaikjnpiTJsesa  io  etai'LiaiE 
aitemaL'teff  fcirae^sf^  andmi'r^iwr  of  .ideniifTeo' n'sAj. 

Spwcilic  <FCial  and  Praclice  Suminarjr 

KM. 

!ji'  M  km  !Sc.i:n"iimJ 

SI'  li  IMh  km  i^^nalaii 
ai-  Ll  bitaiU  ■  Klic  Nn^ni-I  Sli^p 
^  7  (liaiV  n!  A-iJvJB  raia 
Knot  IK  III 

ai'ti  tnknta  LMViUc.  inl  nkifln  IK  iti 
!Kjl3.UI^Hb*a 

ai-3  1  LMsglQitNQfrtlEnl^Mii 
3i-i±  iT^mf^HMIrilt^ni-iiii 

Specific  Praclice»  by  Goal 


5G 1 _ Pr9-]Kara  Tot  Risk  ManaganiBnl 

P.reperairon  for  ris^r  rransgerr-eni  is  comSucied. 

Pneparalion  i&  ccnduoiiBC  by  EstabliEhing  and  maindaining  a  sirabegy  for 
identifying,  anayzing,  ano  nnjtgaJting  riEke.  ~hiE  iElypcaly  cccjmEnled 
in  a  riak  rranagEnent  par.  The  risk  nranagenent  Btralegy  addreESES 
the  speofio  actenE  and  rranagenent  approach  used  to  apply  arc 
omlr-ol  the  risk  nanagEmentprograrrL  Th  s  incudes  idEnflirying  the 
so'jrcES  of  risk;  the  scherre  used  to  cat^orizE  risks;  and  the 
paraneters  used  to  evaualE,  bound,  and  contro  risks  for  effePtve 
handling. 

SP  1.1  DetBrmlriB  Risk  Sources  and  CstsgorleB 

Dere.TTime  rrs.k  ^ou.nces  and  Gazego.'-ies. 

denlificalion  of  risk  Ecurces  prouides  a  bass  for  systemalically 
ejtaminng  changing  EitualiDnsovEr  tme  to  uncover  cincumstancee  thal 

Risk  WanagBrnenl  (RSKM|  431 


•  Goals  are  required,  so... 

Make  each  specific  and  generic  goai 
in  CMMI  into  a  policy  statement 


Risk  Manaaement 

Policy  1 

Projects  shall  conduct 
preparation  for  risk 
management. 

Projects  shall  identify  and 
analyze  risks  to  determine  their 
reiative  importance. 

Policy  3 

Projects  shall  handle  and 
mitigate  risks  are  handled  and 
mitigated,  where  appropriate,  to 
reduce  adverse  impacts  on 
achieving  objectives. 

Policy  4 

Projects  shall  institutionalize 

Risk  Management  as  a  defined 
process. 

_ _ F 
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Constructing  Policies  -  Option  2 


MORTHROP  CRUMMJXN 


Defining  the  future 


’■■mii'ij; 

AJlhough  Ihe  prinar/  EnphasiE.  of  Ihs  Risk.  ManagEnsnt  procBSE  arBa 
is  on  (hiE  pr-ojecL  ihe  ocncepts  can  alsc  be  app  ed  to  manage 
crpanizat onal  rsks. 


Related  Procga-a  Areas 


.RE.feriioi  ihe  PianningpriDceM  anaa  forwi:^&  }nformarKX\  atetri 

ideniilf\Gat<on  Qfp«jeci-  fisks  and  pfann^  Jbr  jivoSemenr  of  rei'evani 
arafce^rciyens. 

ReJer  io  ihe  F^^^iecr  h'fani.itar^  ano'  Owrirai  process  an&a  for  more 
ih/ainahcin  aiour  JTiom'ronng  projeci  risks. 

ReJer  io  ihe  Dscisktn  Analysis  ano'  Reaoiufi'on  prooess  area  Asr  mere 
infainah'on  aiourijiamg  a  .formal' evaJoaSonpran^esa  io  etai'LiaiE 
aitemaL'teff  fcirae^sf^  andmi'r^iwr  of  .ideniifTeo' n'afcff. 

Spwcilic  <FCial  and  Praclice  Suminarjr 

KM. 

!ji'  M  km  !Sc.i:n"iimJ 

SI'  li  IMh  km  i^^nalaii 
ai-  Ll  bitaiU  ■  Klic  Nn^ni-I  Sli^p 
^  7  (liaiV  n!  A-iJvJB  raia 
Knot  IK  III 

ai'ti  tnknta  LMViUc.  inl  nkifln  IK  iti 
!Kjl3.UI^Hb*a 

ai-3  1  LMsglQitNQfrtlEnl^Mii 
3i-i±  iT^mf^HMIrilt^ni-iiii 

SpwFfTc  Praclic»  by  Goal 


5G 1 _ Pr9-]Kara  Tot  Risk  ManaganiBnl 

P.reperairon  for  ris^r  rransgerr-eni  is  comSucied. 

Pneparalion  i&  ccnduoiiBC  by  EstabliEhing  and  maindaining  a  sirabegy  for 
idendiying,  anayzing,  anc  nnjtgaJting  riEke.  ~hiE  iElypcaly  cccjmEmed 
in  a  riak  rranagEnent  par.  The  risk  nranagenent  Btralegy  addreESES 
the  speofio  actenE  and  roanagenent  approach  used  to  apply  arc 
omlr-ol  the  risk  nanagEmentprograrrL  Th  s  incudes  idErdirying  Ihe 
so'jrcES  of  risk;  the  scherre  used  to  cat^orizE  risks;  and  Ihe 
parameters  used  to  evaualE,  bound,  and  contro  risks  ibr  effePtve 
handling. 

SP  1.1  DetBrmlriB  Risk  Sources  and  CatsgorleB 

Dere.TTime  rrs.k  ^ou.nces  arref  carego.fies. 

denlificalion  of  risk  Ecurces  prouides  a  bass  for  systemalically 
ejtaminng  changing  BitualionsovEr  tmE  to  uncover  cincumstancee  thal 

Risk  WanagBrnenl  (RSKM|  431 


•  Practices  are  expected,  SO... 
Make  each  specific  and  generic 
practice  in  CMMI  into  a  policy 
statement 


\  Risk  Management 

—  \  Policy  1  Projects  shall  determine  risk 
\  sources  and  categories, 

j  Policy  2  Projects  shall  define  the 
/  parameters  used  to  analyze  and 

_ /  categorize  risks, 

/  Etc. 

r _ ^ 

•  Since  practices  are  only  expected, 
must  create  an  opportunity  for  the 
unexpected  -  a  deviation! 

-  Does  the  approach  still  meet  the 
CMMI  goal? 
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Process  (Description) 


MORTHROP  CRUMMJXN 


Defining  the  future 


"A  documented  expression  of  a  set  of  activities  performed  to  achieve 
a  given  purpose.  A  process  description  provides  an  operational 
definition  of  the  major  components  of  a  process.  The  description 
specifies,  in  a  complete,  precise,  and  verifiable  manner,  the 
requirements,  design,  behavior,  or  other  characteristics  of  a  process.' 


-  Glossary,  CMMI-DEV  vl.2 


•  Processes  describe  the  steps  to  be  taken 

-  Typical  process  established  in  the  organizational  standard  process 

-  Tailored  by  the  project  to  fit  their  needs 
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Using  Processes 


MORTHROP  CRUMMJXN 


Defining  the  future 


GP  3.1  Establish  a  Defined  Process 

Establish  and  maintain  the  description  of  a  defined  process. 

•  "Defined  process"  means  tailored  from  an 
organizational  standard  process 

-  Both  projects  and  functional  groups  must  tailor 

•  The  detail  of  the  processes  is  driven  by  the 
similarities  between  project  needs 

-  If  projects  are  similar,  one  size  fits  all 

-  The  more  your  project  is  different  than  the  typical  project 
In  the  organization,  you  more  tailoring  you  need 

•  Tailoring  does  not  require  approval 

-  Policies  already  define  the  acceptable  limits  (i.e.,  tailor  as 
much  as  desired  as  long  as  you  don't  violate  policy) 
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Constructing  Processes 


Typical  attributes  of  each 
process  element  (per  CMMI ) 

-  Process  roles 

-  Applicable  standards 

-  Applicable  procedures, 
methods,  tools,  and  resources 

-  Process-performance  objectives 

-  Entry  criteria 

-  Inputs 

-  Product  and  process  measures 
to  be  collected  and  used 

-  Verification  points  (e.g.,  peer 
reviews) 

-  Outputs 

-  I  nterfaces 

-  Exit  criteria 
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MORTHROR  CRUMMJXN 


Constructing  Processes 


OHa-bir  tihihapan 
’■■mii'ij; 

AJlhough  Ihe  prinar/  EnphasiE.  of  Ihs  Risk.  ManagEnsnt  procBSE  arBa 
is  on  (hiE  pr-ojecL  ihe  ocncepts  can  alsc  be  app  ed  to  manage 
crpanizat onal  rsks. 


Related  Procga-a  Areas 


.RE.feriioi  ihe  PianningpriDceM  anaa  forwi:^&  }nformarKX\  atetri 

ideniilf\Gat<on  Qfp«jeci-  fisks  and  pfann^  Jbr  jivoSemenr  of  rei'evani 
arafce^rciyens. 

ReJer  io  ihe  F^^^iecr  h'fani.itar^  ano'  Owrirai  process  an&a  for  more 
ih/ainahcin  aiour  JTiom'ronng  projeci  risks. 

ReJer  io  ihe  Dscisktn  Analysis  ano'  Reaoiufi'on  process  area  Asr  more 
infainah'on  aiourijiamg  a  .formal' evaJoaSonpiosesa  io  etai'Liars 
aitemaL'teff  fcirae^sf^  andmi'r^iwr  of  .ideniifTeo' n'sAj. 

Spwcilic  <FCial  and  Praclice  Suminarjr 

KM. 

!ji'  M  IM^itMi  km  !Sc.i:n"iimJ 

SI'  li  IMh  km  i^^nalaii 

ai-  Ll  bitaiU  ■  Klic  Nn^ni-I  Sli^p 
^  7  (liaiV  n!  A-iJvJB  raia 
Knot  IK  III 

ai'ti  tnknta  LMViUc.  inl  nkifln  IK  iti 
!Kjl3.UI^Hb*a 

ai-3  1  LMsglQitNQfrtlEnl^Mii 
3i-i±  iT^mf^HMIrilt^ni-iiii 


SpwFfTc  Praclic»  by  Goal 


5G 1 _ Pr9-]Kara  Tot  Risk  ManaganiBnl 

P.reperairon  for  ris^r  rransgerr-eni  is  comSucied. 

Pneparalion  i&  ccnduoiiBC  by  EstabliEhing  and  maintaining  a  sirabegy  for 
identifying,  anayzing,  ano  nnjtgaJting  riEke.  ~hiE  iElypcaly  ccojmEnted 
in  a  riak  rranagEnent  par.  The  risk  nranagenent  Btralegy  addreESES 
the  speofio  actcnE  and  rranagenent  approach  used  to  apply  arc 
omlr-ol  the  risk  nanagEmentprograrrL  Th  s  incjdes  idEnlifying  the 
so'jrcES  of  risk;  the  scherre  used  to  cat^orizE  risks;  and  the 
paraneters  used  to  evaualE,  bound,  and  contro  risks  for  effePtve 
handling. 

SP  1.1  DetBrmlriB  Risk  Sources  and  CsEsgorleB 

Dere.TTime  rrs.k  ^ou.nces  and  Gazego.'-ies. 

denlificalion  of  risk  Ecurces  prouides  a  bass  for  systemalically 
ejtaminng  changing  EitualiDnsovEr  tme  to  uncover  cincumstancee  thal 

Risk  WanagBrnenl  (RSKM|  431 
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Option  1 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  Practices  are  expected,  SO... 
Make  each  specific  and  generic 
practice  in  CMMI  into  a  process 
description  step 


\  Risk  Management 

\  Step  1  Project  determines  risk  sources 

\  and  categories, 

j  Step  2  Project  defines  the  parameters 

/  used  to  analyze  and  categorize 

/  risks, 

/  Etc. 

r  I _ ^ 


•  Tailoring  may  create  a  problem  in 
meeting  the  goal 


Constructing  Processes 


OHa-bir  tihihapan 
’■■mii'ij; 

AJlhough  Ihe  prinar/  EnphasiE.  of  Ihs  Risk.  ManagEnsnt  procBSE  arBa 
is  on  (hiE  pr-ojecL  ihe  ocncepts  can  alsc  be  app  ed  to  manage 
crpanizat onal  rsks. 


Related  Procga-a  Areas 


.RE.feriioi  ihe  PianningpriDceM  anaa  forwi:^&  }nformarKX\  atetri 

ideniilf\Gat<on  Qfp«jeci-  fisks  and  pfann^  Jbr  jivoSemenr  of  rei'evani 
arafce^rciyens. 

ReJer  io  ihe  F^^^iecr  h'fani.itar^  ano'  Owrirai  process  an&a  for  more 
ih/ainahcin  aiour  JTiom'ronng  projeci  risks. 

ReJer  io  ihe  Dscisktn  Analysis  ano'  Reaoiufi'on  prooess  area  Asr  mere 
infainah'on  aiourijiamg  a  .formal' evaJoaSonpran^esa  io  etai'LiaiE 
aitemaL'teff  fcirae^sf^  andmi'r^iwr  of  .ideniifTeo' n'afcff. 

Spwcilic  <FCial  and  Praclice  Suminarjr 

KM. 

!ji'  M  IM^itMi  km  !Sc.i:n"iimJ 
SI'  li  IMh  km  i^^nalaii 
ai-  Ll  bitaiU  ■  Klic  Nn^ni-I  Sli^p 
^  7  (liaiV  n!  A-iJvJB  raia 
Knot  IK  III 

ai'ti  tnknta  LMViUc.  inl  nkifln  IK  iti 
!Kjl3.UI^Hb*a 

ai-3  1  LMsglQitNQfrtlEnl^Mii 
3i-i±  iT^mf^HMIrilt^ni-iiii 


Specific  Praclice»  by  Goal 


5G 1 _ Pr9-]Kara  Tot  Risk  ManaganiBnl 

P.reperairon  for  ris^r  rransgerr-eni  is  comSucied. 

Pneparalion  i&  ccnduoiiBC  by  EstabliEhing  and  maindaining  a  sirabegy  for 
idendiying,  anayzing,  anc  nnjtgaJting  riEke.  ~hiE  iElypcaly  cccjmEmed 
in  a  riak  rranagEnent  par.  The  risk  nranagenent  Btralegy  addreESES 
the  speofio  actenE  and  roanagenent  approach  used  to  apply  arc 
omlr-ol  the  risk  nanagEmentprograrrL  Th  s  incudes  idEntirying  the 
so'jrcES  of  risk;  the  scherre  used  to  cat^orizE  risks;  and  the 
parameters  used  to  evaualE,  bound,  and  contro  risks  ibr  effePtve 
handling. 

SP  1.1  DetBrmlriB  Risk  Sources  and  CatsgorleB 

Dere.TTime  rrs.k  ^ou.nces  arref  carego.fies. 

denlificalion  of  risk  sources  provides  a  bass  for  systemalically 
ejtaminng  changing  EitualiDnsovEr  tmE  to  uncover  cincumstancee  thal 

Risk  WanagBrnenl  (RSKM|  431 
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Option  2 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  I  f  more  detail  is  desired,  add 
subpractices 

Risk  Management 

Step  1  Project  determines  risk  sources, 

i  Step  2  Project  determines  risk 

\  categories. 

—  \  Step  3  Project  defines  consistent  criteria 
\  for  evaluating  and  quantifying 

)  risk  likelihood  and  severity  risks. 

/  Step  4  Project  defines  thresholds  for 
— I  /  each  risk  category. 

/  Step  5  Project  defines  bounds  on  the 

'  extend  to  which  thresholds  are 

applied  against  or  within  a 
category. 

I  F 


•  Note:  subpractices  only  represent 
one  way  practices  might  be  met 


Constructing  Processes  - 


Typical  attributes  of  each 
process  element  (per  CMMI ) 

-  Process  roles 


-  Applicable  standards 

-  Applicable  procedures,  methoc 
tools,  and  resources 

-  Process-performance  objective 

-  Entry  criteria 


s, 


-  Inputs 

-  Product  and  process  measures  to 
be  collected  and  used 


-  Verification  points  (e.g.,  peer 
reviews) 

-  Outputs 

-  I  nterfaces 


-  Exit  criteria 
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Considerations 


MORTHROP  CRUMMJXN 


Defining  the  future 


Risk  Management 

Step  1  Project  manager  determines  risk 
sources. 

Step  2  Project  will  use  the  XXX  risk 
categories. 

Step  3  Project  defines  consistent  criteria 
for  evaluating  and  quantifying 
risk  likelihood  and  severity  risks 
in  the  Risk  Management  Plan. 

Step  4  Project  defines  thresholds  for 
each  risk  category. 

Step  5  Project  defines  bounds  on  the 
extend  to  which  thresholds  are 
applied  against  or  within  a 
category  as  per  procedure  YYY. 


Etc. 


GP  2.2 


MORTHROP  CRUMMJXN 


Defining  the  future 


GP  2.2  Plan  the  Process 

Establish  and  maintain  the  plan  for  performing  the  process. 


•  Plan  =  description  of  activities  +  budget  + 
schedule 

-  Description  of  activities  is  addressed  in  GP  3.1  (process 
description) 

-  Budget  is  addressed  in  GP  2.3;  resources  in  GP  2.4 

•  Schedules  for  some  process  areas  may  be  tied  to 
program  events 

-  E.g.,  DAR  events  may  not  be  separateiy  shown  on  a  scheduie, 
but  pians  shouid  make  ciear  the  conditions  under  which  a  DAR  is 
to  be  conducted 
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Documenting  Choices  in  Plans 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  Policies  identify  what  must  happen 

•  Process  descriptions  and  procedures  describe  the 
steps  to  be  performed 


•  Plans  describe  how  the  process  is  instantiated 


Process  Asset  Library 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  The  organization's  process  asset  library  is  a 
collection  of  items  maintained  by  the  organization 
for  use  by  the  people  and  projects  of  the 
organization 

-  Organizational  policies 

-  Defined  process  descriptions 

-  Procedures 

-  Development  plans 

-  Acquisition  plans 

-  Quality  assurance  plans 

-  Training  materials 

-  Process  aids  (e.g.,  checklists) 

-  Lessons-learned  reports 
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Keys  to  Quickly  Establishing  an  Effective 
PAL 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  Section  1  -  Organizational  materials 

-  Policies,  processes,  procedures,  templates,  tools,  etc. 

-  Provides  central  access  to  all  projects 

-  "Blessed"  by  the  process  group 

•  Section  2  -  Project  examples 

-  Plans,  tailored  processes,  specs,  etc. 

-  Provides  examples  -  helps  some  visual  the  desired  state 

-  Submitted  by  the  projects  at  their  own  discretion,  or  as 
identified  by  the  process  group 

•  Eventually... 

-  Process  group  can  "bless"  best-in-class  examples 

-  Good  examples  can  be  turned  into  templates 
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Measures 


MORTHROP  CRUMMJXN 


Defining  the  future 


*  The  CMMI  discusses  measures  in  several  ways 

-  PMC  SP  1.1:  Monitor  the  actual  values  of  the  project  planning 
parameters  against  the  project  plan. 

(estimates  of  Work  Product  and  Task  Attributes,  effort,  cost) 

-  GP  2.8:  Monitor  and  control  the  process  against  the  plan  for 
performing  the  process  and  take  appropriate  corrective  action, 
(activities  vs.  plan,  achievements  vs.  schedule,  effort  vs.  budget) 

•  The  Measurement  &  Analysis  process  area  suggests  that 
measurement  system  be  defined,  but  does  not  specify  measures 
which  must  be  used 


SG  1  Align  Measurement  and  Analysis 
Activities 

SP  1.1  Establish  Measurement  Objectives 
SP  1.2  Specify  Measures 
SP  1.3  Specify  Data  Collection  an6  Storage 
Proce(dures 

SP  1.4  Specify  Analysis  Procedures 


SG  2  Provide  Measurement  Results 

SP  2.1  Collect  Measurement  Data 
SP  2.2  Analyze  Measurement  Data 
SP  2.3  Store  Data  and  Results 
SP  2.4  Communicate  Results 
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Keys  to  Quickly  Establishing  an  Measures 


MORTHROP  GRUMMAN 


Defining  the  future 


•  Section  1  -  Organizational-wide  measures 

-  Focus  on  enabling  future  projects  to  estimate  based  on  past  projects 

-  Common  Work  Breakdown  Structure  (or  mapping  to  one) 

-  Effort  expended,  by  WBS  element  (all  time  accounting) 

-  Size,  characteristics  of  the  project,  product 

-  Clear  operational  definitions  of  the  base  measures 

-  Capture  the  measures  in  an  organizational  measurement  repository 

•  Section  2  -  Project-specific  measures 

-  Identify  (but  don't  collect)  the  project-specific  measures  used  (e.g., 
customer  dictated  metrics) 

•  Eventually... 

-  Add  organizational-wide  metrics  as  you  see  the  need  or  opportunity 

-  Consider  collecting  metrics  to  allow  the  organization  to  calibrate  a 
cost  estimation  model  (e.g.,  COCOMO,  COSYSMO) 

-  Be  patient! 


31 


Measurement  Repository 


MORTHROP  CRUMMJXN 


Defining  the  future 


organization's  measurement  repository  -  A  repository  used  to  collect 
and  make  available  measurement  data  on  processes  and  work  products, 
particularly  as  they  relate  to  the  organization's  set  of  standard  processes. 

This  repository  contains  or  references  actual  measurement  data  and  related 
information  needed  to  understand  and  analyze  the  measurement  data. 

-  Glossary,  CMMI-DEV 


•  I  nitial  focus  in  on  supporting  estimation 

-  Effort  expended 

-  Product  size  and  other  attributes 

-  Project  characteristics 

•  Later... 

-  Quality  measures 

-  Statistical  management  data,  causal  analysis  data 
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Training 


MORTHROP  CRUMMJXN 


Defining  the  future 


Purpose 

-  Develop  the  skills  and  knowledge  of 
people  so  they  can  perform  their  roies 
effectively  and  efficiently 


Key  actions 

-  Identifying  the  training  needed  by  the  organization 

-  Obtaining  and  providing  training  to  address  those  needs 

-  Establishing  and  maintaining  training  materials 

-  Establishing  and  maintaining  training  records 

-  Assessing  training  effectiveness 
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Training  Scope 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  Skills  and  knowledge  may  be: 

-  Technical  -  ability  to  use  the  equipment,  tools,  materials,  data,  and 
processes 

-  Organizational  -  behavior  within  and  according  to  the  employee's 
organization  structure,  role  and  responsibilities,  and  general 
operating  principles  and  methods 

-  Contextual  -  self  management,  communication,  and  interpersonal 
abilities  needed  to  successfully  perform  in  the  organizational  and 
social  context  of  the  project  and  support  groups 

•  Training  options 

-  Classroom  training 

-  Web-based  training 

-  Guided  self  study 

-  Formalized  on-the-job  mentoring 
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Is  the  Staff  Qualified  to  Do  Their  Work? 


MORTHROP  CRUMMJXN 


Defining  the  future 


An  organizational  responsibility! 


•  What  are  the  minimum 
skills  and  knowledge 
needed  to  perform  their 
job  function? 

•  Does  each  individual 
possess  these  skills? 

-  If  not,  training  is  expected 
to  address  the  gaps 


How  does  the  organization  maintain 
a  skilled  and  knowledgeable  workforce? 
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Strategies  for  Organizational  Training  -  1 
of  2 


MORTHROP  GRUMMAN 


Defining  the  future 


•  start  by  defining  the  key  job  functions  in 
the  organization 

-  E.g.,  project  manager,  software  engineer, 
quality  assurance  specialist 

•  I  dentify  the  requisite  knowledge  associated  with  each  function 

•  Define  a  set  of  course  modules  that  impart  this  knowledge 

-  Map  modules  to  job  functions 

-  Some  modules  will  be  common  to  multiple  job  functions 

•  Acquire  training  materials  and  trainers 

-  Should  reflect  the  organization's  policies  and  processes 

-  Unlikely  that  standard  vendor/university  courses  will  fit 

•  Ensure  all  the  CMMI  process  areas  are  addressed 

-  Knowledge  needed  to  perform  the  process,  NOT  a  course  about  the 
CMMI  requirements  for  that  process  area 

-  Include  performers  of  the  process,  and  those  supporting 
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Strategies  for  Organizational  Training  -  2 
of  2 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  I  dentify  each  employee  by  their  job  function(s),  map  to 
required  courses 

-  If  the  employee  already  has  the  identified  minimum  knowledge,  they 
do  not  need  to  take  the  course 


•  Establish  student  records 

-  Who  has  completed  what  course,  waivers 


•  Review  required  training  with  employees 

-  Career-planning,  promotions,  new  hires 


Where  additional  project-specific  training 
is  required  (e.g.,  tools,  methods),  adopt  a 
similar  approach  at  the  project  level 

-  Project  Planning  SP  2.5 
addresses  project  specific  training 
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Summary 


MORTHROP  CRUMMJXN 


Defining  the  future 


•  An  organization  adopting  the  CMMI  modei  has  to 
make  numerous  decisions: 

-  Scope  of  the  improvement  effort 

-  Model  representation 

-  I PPD  extension 

-  Structure  of  the  policies  and  processes 

-  Training  program 

-  Measurement  repository 

-  Etc. 
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•  These  choices  made  have  a  profound  effect  on  the 
value  of  the  improvements,  buy-in  of  the  organization, 
and  the  ultimate  success  of  the  CMMI  effort 


Continentol  Reoching  Solutions 


Achieving  Quality  QPPO  via  Effective 
Usage  of  PPBs  and  PPMs 


Dr.  Bin  Cong 

SEI  Certified  High  Maturity  Lead  Appraiser 
SEI  Certified  Intro  to  CMMI  Instructor 

CRS  Tech 

Professor  and  Director 
Master  of  Software  Engineering 
Cal  State  Univ.  at  Fullerton 
bcong@fullerton.edu 


Outline 


•  PPBs  and  PPMs’  usage  in  quality  goal  setting 

•  PPMs  and  PPBs’  usage  in  quality  goal  management 

•  Controllable  factors 
Improvement  Observed 


Some  lessons  learnt 


The  Context  of  the  Case  Studies 


•  Org  is  serving  one  eustomer 

•  High  quality  is  the  most  Important  Produet 
Requirement 

•  Business  goals  are  set  up  by  the  elient 


Customer’s  Product  Quality  Requirement 


•  4  Nines  -  99.99%: 


Escaped  defects  <0.1  per  KLOC 


Org’s  Quality  Objective 


•  Defects  density  identified  in  acceptance  test  is  less  than 
O.ll/KLOC  which  is  based  on  the  AT  performance  baseline. 


■  Historical  data  shows  that  the  lower  bug  rate  identified  by  aeceptance  test,  the  lower  of 
delivered  bug  rate.  With  95%  confidence,  it  has  been  show  that  if  the  acceptance  test  bug 
rate  lower  than  O.ll^/KLOC,  delivered  bug  rate  will  be  lower  than  O.l^/KLOC. 


The  Rationale  for  Choosing  the  Quality  Objective 


•  It  meets  clients’  quality  requirement. 

•  Org’s  baseline  supports  it. 

•  The  org’s  metrics  support  it. 

•  It  can  be  easily  used  by  project  team. 


Setting  up  the  Interim  Quality  Objectives 


•  The  following  quality  control  activities  are  conducted 
before  the  acceptance  test  is  performed  by  the  independent 
Testing  Center: 

-  Requirement  Peer  Review 

-  System  Design  Peer  Review 
-Detailed  Design  Peer  Review 

-  Code  Inspection  +  Unit  Test 

-  System  Test 


The  related  interim  goals  need  to  be  developed  to  ensure 
achieving  the  Quality  Objective,  thus  the  goal  becomes  a 
manageable  one. 


PPBs  Needed  to  Support  the  Interim  Goals 


•  Defect  injection  distribution 

•  Defect  removal  rate  in 
requirement/design/code  review  +  UT  and 
system  test 

•  Efforts  devoted  to  these  quality  control 
activities 


Abnormal  Analysis 


Effort  baselines  is  needed  to  support  this  analysis 
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Quality  Related  Baselines  —  Measured  by  defect  removal  rate 
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Quality  Related  QPPOs 


Acceptance  test  bug  rate  lower  than  0.11  defects/KLOC : 

(DRequirement  review  identifies  at  least  0.09*  total 
number  of  estimated  defeets ; 

©System  design  review  identifies  at  least  0.1*  total 
number  of  estimated  defeets ; 

©Detail  design  review  identifies  at  least  0.02*  total 
number  of  estimated  defeets ; 

©Code  Review  and  UT  identifies  at  least  0.36*  total 
number  of  estimated  defeets ; 

©System  test  identifies  at  least  0.41*  total  number  of 
estimated  defeets. 


Another  Example 


■  Requirement  Peer  Review  should  at  least  identify 
80%  of  defects  introduced  so  far 

■  Design  Peer  Review  should  at  least  identify  70%  of 
remaining  defects  introduced  so  far 


■  Code  Inspection  should  at  least  identified  40%  of 
remaining  defects  introduced  so  far 


System  Testing  should  at  least  identify  90% 
remaining  defects  introduced  so  far 


Interim  Goals  and  Overall  Quality  Objective 


•  Statistical  studies  show  that  if  the  Interim 
Goals  are  achieved,  the  overall  goal  will  be 
achieved  too. 

•  QPM  is  all  about  managing  the  goal 
achievement. 


Prediction  models  needed  for  quality  goal  management 


•  Number  of  defects  introduced  in  Requirement  Phase 

•  Number  of  defects  introduced  in  Design  Phase 

•  Number  of  defects  introduced  in  Coding  Phase 

•  Number  of  defects  removed  by  Requirement  Peer  Review 

•  Number  of  defects  removed  by  Design  Peer  Review 

•  Number  of  defects  removed  by  Code  Review  for  Java  and 
•Net 

•  Number  of  defects  removed  by  Code  Review  for  C  and  C++ 

•  Number  of  defects  removed  by  System  Test 

•  Gompertz  Model  -  a  Reliability  Growth  Model 


Risk  Management 


Monte  Carlo  is  used  for  managing  risks  in 
obtaining  Quality  Goals  during  the  planning 
phase  and  throughout  the  LC. 


Relationship  between  Goals  and  Key  Subprocesses 
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Critical  Key  Sub-process  Selection  Criteria 


■  Customer’s  concerns 

■  The  Impact  to  the  QPPOs 

■  Statistical  impact  analysis 


10.000  Trials 


Contribution  to  Variance  V\ev-f 


10.000  Trials 


Contribution  to  Variance  View 
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The  Goal-Model-Baseline  Matrix 


How  Models  fit  in  the  Quality  Goal  Mgt 
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It  is  all  about  achieve  the  goals! 


Risks,  Issues,  and  Corrective/Preventive  Actions 


Overview  on  How  PPBs  and  PPMs  are  Used 


2009-11-18 
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Probability 


Monte  Carlo  Simulation  on  Goal  Achievement 


100.000  Trials 


Frequencv  View 


Controllable  Factors 


Sources  of  variation 

HM  means  you  truly  understand  your  critical 
processes. 

Where  you  might  make  adjustments 
Key  areas  to  improve  your  process 


Which  model  allows  you  to  adjust? 


•  Defect  Removal  Predictive  Model  for 
Requirement  Peer  Review: 

f  (Size,  Type,  Complexity) 


f(Size,  Review  Effort,  Review  Team  Ability 
Index,  Type) 


Improvements  Observed 


Defect  removal  rate  improved  in  code  review  and  UT«&;IT. 


Defect  removal  Rate(Y2007-Y2008) 


Code  Review  UT&IT  Peer  Review  Peer  Review  Peer  Review 

(KLOC)  (Per  FP)  (Technial  (RA, Per  page)  (HD, Per  page) 

Solution,  Per  page) 


percentage  of  DRR 


Defect  Removal  Pattern  Moves  to  Front 


Defect  Removal  Rate  (Y2007-Y2008) 
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Some  Lessons  Learnt  - 1 


•  Set  up  the  big  picture  first  with  clearly  defined  overall  goals 
and  interim  goals. 

•  Clearly  think  through  how  the  PPBs  and  PPMs  will  be 
used.  You  may  want  to  write  the  PPBs  and  PPMs’  User 
Guidelines  before  actually  developing  them.  The  PPBs  and 
PPMs  will  be  refined  from  time  to  time  but  how  they  are 
used  will  change  much  less  frequently. 

•  Model  development  process  is  to  really  get  to  know  your 
process:  factors  in  the  model  -  sources  of  variations.  It  is 
not  enough  if  you  only  master  the  statistical  techniques  and 
know  how  to  use  Minitab. 

•  Model  development  process  can  also  help  you  to  identify 
areas  to  improve. 


Some  Lessons  Learnt  -  II 


•  When  conducting  regression  analysis,  do  not  just  look  at  R 
square  but  also  think  “will  the  model  allow  you  do  What-If 
analysis?” 

•  Benchmarking  a  process  does  not  make  it  a  key  process.  A 
key  process  should  also  be  the  focus  of  your  improvement. 
The  factors  in  a  good  process  performance  model  are  the 
candidate  areas  to  improve. 

•  PPBs  can  support  the  use  of  Monte  Carlo  simulation. 

•  Spec  limits  and  control  limits  can  get  people  confused. 

•  QPPOs  and  Controllable  F actors ! ! ! 


Thank  you 


Army  ERP  Center  of  Expertise 


Tailoring  CMMI  for  an  Enterprise  Resource 
Planning  COTS  Software  Environment 


Director,  Business  Transformation  and  E- Systems  Directorate, 
Weapons  &  Software  Engineering  Center 
Alison  L  Schwier 
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Mr.  George  Albinson 


Agenda 

>  Background 

>  SEI  CMMI  models 

>  COE  Decision  to  use  CMMI  for  Development 

>  Tailoring  Armament  Software  Engineering 
Center  Policies/Procedures 


>  Lessons  Learned 


Consolidated  ERP  Integration  Strategy  Memo 

Dated  7  Oct  08 


1.  A  decision  was  made  by  the  Army  Business  Mission 
Area  (BMA)  Executive  Board  at  its  meeting  on  September 
26,  2008  to  approve  the  plan  for  a  transition  from  the 
I  current  federated  ERP  integration  path,  to  a  combined 
ERP  post-2011.  The  decision  is  consistent  with  direction 
from  the  OSD  BTA  and  the  Defense  Acquisition  Executive. 


■  4.  It  is  critical  as  the  Army  moves  to  an  automated 

logistics  program  and  a  clean  financial  audit,  on  the  path 
I  to  broader  total  asset  and  resource  visibility,  that  the 
transition  from  federated  to  integrated  ERPs  be 
deliberate,  effective,  cost-aware  and  rapid.  We  require  a 
I  clear  framework  for,  not  only  the  consolidated  ERP  effort, 
but  also  the  broader  management  of  business  processes 
and  the  associated  information  technology  systems. 
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Governance  -  Org  Chart 


Integrated  Deployment  &  Sustainment 


Users 


Functional 
Cells 


Technology 

Cells 


Army  ERP  COE  Concepts 


An  Army  in-house  system  integrator 

-  ''true''  ERP  with  full  cross-Domain  business  processes  using  SAP  ERP  based  on 
Business  Process  Reengineering 

-  full  scale  in-production  platform  with  landscapes  capable  of  launching  COTS 
prototypes  with  vetted  best  practices  &  lessons  learned 

-  technical  risk  reduction  &  cost  mitigation  capabilities  &  techniques  for  rapid, 
effective  &  efficient  implementations 

-  value  added  stakeholder  relationships 

•  Strong  ERP  vendor  relations 

❖ARDEC  sec  CMMI  Level  5  center  of  excellence  using  in-house  resources  & 
lean  tools 

•  Defense  Ammunition  Center  Trainers 

•  OSD  Business  Transformation  Agency  ESG  Member 

•  Other  Service  ERP  PM's  in  Navy  &  DLA 


AMC  Path  Forward  on  Army  ERP  COE 

Today's  Realities 

Large  ERP  Programs  don't  die 

-  Sunk  costs 

-  Stakeholder  resistance 

-  System  Integrator  constituencies 

-  Belief  that  government  can't  do  the  heavy  lifting 
Army  In-sourcing 

-  Some  view  AMC  as  large  inflexible  lethargic  bureaucracy 

-  AMC  has  the  SME  talent 

-  AMC  has  the  base  to  start  now  &  grow  technology  from  SEC  &  eNOVA 

-  PEO  EIS  has  the  acquisition  capabilities 

-  Strategic  move  from  requirements  analysis  to  conference  room  pilot 
approval 

-  Move  from  DoD  5000  driven  System  Integrator  contract  milestones  to 
moving  at  Commander's  pace  with  latest  technologies  &  latest  strategies 

•  Slices  of  end  to  end  processes  versus  large  monolithic  stove  pipe  implementations 

•  Build  to  holistic  enterprise  that  matches  strategy  to  transactions  versus  huge  integration 
costs  among  internally  focused  stove  pipes 


"The  Americans  will  always  do  the 
right  thing...  after  they've 
exhausted  all  the  alternatives."  -- 
Winston  Churchill 


Understand  SEI's  CMMI-DEV  and  CMMI-SVC 


i 

>vc 

Provides  a 

comprehensive  set  of 
best  practices  for 
providing  services. 


ACQ 


Guidance  for 
application  of  CMMI 
best  practices  by  the 
acquirer 


In  those  cases  where 
the  service  system  is 
large  and  complex,  the 
CMMl’DEV  mode]  can 
be  effectively  used  to 
develop  such  a  system, 


Focuses  on  the 
activities  of  the  service 
provider 


In  those  cases  where 
the  acquirer  has  a  role 
in  product  or  service 
development,  CMMI- 
OEV  should  also  be 
used 


Focuses  on  process 
improvement  In 
development 
organization's 


The  COE  is  following  the  CMMI-DEV  model  -  to  develop  this  large  and  complex  ERP  system 


Hybrid  CMMI  Implementation  for  Army  ERP 


CMMI-SVC 


Initial  large  and  complex 
development  effort;  augment  with 
CMMI-SVC  constellation 


Appraise  to 
CMMI-DEV 


Eventually:  Maintain  a  high-quality 
service-providing  organization 


CMMI  Model  Comparison 


CMMI-SVC  CMMI-DEV  CMMI-ACQ 


Process 

Areas 

CMMI  Model 
Foundation 

y^ 

Services  Specific 

^2 

Engineering 

Acquisition 

y/^ 

^  Change  Management 
added  by  COE  to  cover 
this  crucial  COTS  function 


^  9  CMMI-DEV  Model  Foundation  Process  Areas  have 
demands  specific  to  COTS: 

•  Project  Planning  •  Process  and  Product  Quality 

•  Risk  Mgmt  Assurance 

•  Requirement  Mgmt  •  Integrated  Project 

•  Decision  Analysis  &  Resolution  Management 

•  Supplier  Agreement  Mgmt  •  Configuration  Mgmt 

•  Measurement  and  Analysis 


The  7  Services  Specific  Process 


Service  Delivery  (SD) 

^  Service  System  Transition 
(SST)  (Change  Management) 
Strategic  Service  Management 
(STSM) 

Capacity  and  Availability 
Management  (CAM) 


Areas: 

Service  Continuity  (SCON) 
Incident  Resolution  and 
Prevention  (IRP) 

Service  System  Development 
(SSD) 


^Acquisition  constellation  is  a  resource  when  acquiring  COTS  software  (SAP)  and  services  (contractors  with  SAP  skills). 


Which  process  areas  are  covered  by  each  model? 


Continuous  Representation 


COE's  COTS  Implementation  Process  Areas 


CMMI-DEV  process  areas 


Process  Management 

OPF 

Organizational  Process  Focus 

OPD 

Organizational  Process  Definition  +  IPPD 

OT 

Organizational  Training 

OPP 

Organizational  Process  Performance 

OID 

Organizational  Innovation  and  Deployment 

Project  Management 

PP 

Project  Planning 

PMC 

Project  Monitoring  and  Control 

SAM 

Supplier  Agreement  Management 

IPM 

Integrated  Project  Management  +  IPPD 

RSKM 

Risk  Management 

QPM 

Quantitative  Project  Management 

Engineering 

REQM 

Requirements  Management 

RD 

Requirements  Development 

TS 

Technical  Solution 

PI 

Product  Integration 

VER 

Verification 

VAL 

Validation 

Support 

CM 

Configuration  Management  ** 

PPQA 

Process  and  Product  Quality  Assurance 

MA 

Measurement  and  Analysis 

DAR 

Decision  Analysis  and  Resolution 

CAR 

Causal  Analysis  and  Resolution 

Addition  of  1  CMMI-SVC  process  area 


+ 


**  Tailor  CM  Policy  &  Procedures 

—Version  control  and  numbering 
—Product  release  and  delivery 


Service  Specific 

SST  Service  System  Transition 


Relevant  CMMI-ACQ  Process  Areas 


Process  Area 

COE  Examples 

Agreement  Management  (AM) 

The  purpose  of  Agreement 
Management  (AM)  is  to  ensure  that 
the  supplier  and  the  acquirer 
perform  according  to  the  terms  of 
the  supplier  agreement. 

•At  the  highest  level,  the  agreement 
between  the  Army  and  SAP. 

•Each  services  contract  for  SAP 
contractor  assignments. 

Acquisition  Validation  (AVAL) 

The  purpose  of  Acquisition 

Validation  (AVAL)  is  to  demonstrate 
that  an  acquired  product  or  service 
fulfills  its  intended  use  when  placed 
in  its  intended  environment. 

•Audits  and  reviews  of  base  SAP 
software  against  Army  goals  and 
vision. 

•Functional  and  Physical  audits  of 
various  existing  Army  systems  the 

COE  acquires. 

Acquisition  Verification  (AVER) 

The  purpose  of  Acquisition 

Verification  (AVER)  is  to  ensure  that 
selected  work  products  meet  their 
specified  requirements. 

•Verification  that  the  skills  and 
output  artifacts  from  each  contract 
are  as  expected. 

•Verification  that  the  base  SAP 
software  performs  as  expected  and 
support  levels  are  maintained. 

Decision  Analysis  and  Resolution  for  CMMI-DEV 

Model  Choice 


Our  team  considered: 


Credibility 
"Proven  Product" 


Ease  and  Speed  of 

Implementation 

"Leverage" 


"Applying  CMMI  to  a  COTS  adaptation 
is  a  new  effort.  We  will  be  most 
successful  by  using  Armament 
Software  Engineering  Center's  Level-5 
Organizational  Framework" 


"Armament  Software 
Engineering  Center  Level-5 
policies,  procedures  and 
templates  are  based  on 
CMMI-DEV" 


Cost 


"Costs  include  staffing  and 
training  a  new  process 
engineering  group  for 
CMMI-SVC  model" 


strengthen  internal  and  external  relationships 


ARMY  INTERNAL 


i 

n 

Expansion  of  Policies, 
Procedures,  & 
Examples  to  include 
COTS 


Level -5 
Developed 
Practices 


Share  best 
practices/technical 
papers  related  to 
COTS 


What  Organizational  Standard  Processes  did  we 

tailor? 


Glossary 


•  Translation  guidance  for  COTS  terms  to  custom  development  terms 

•  Added  COTS-specific  terms:  Change  Management,  Custonner  Competency 
Center,  Work  Plan 

•  Updated  definitions:  Configuration  Management  Terms,  Traceability  Matrix 


Add  Service  System  Transition  Process  Area 


•  i.e.  Change  Management 

•  Concept  borrowed  from  CMM1-SVC 

•  Users  often  experience  significant  change  in  SAP  installations 

•  New  Policy  and  Procedures  written 


Configuration  Management  Policy  &  Procedures 


•  CCB  operates  differently 

•  COTS  Issue  Management  process  needs  standardization 

•  Audit  processes  are  slightly  different 


Tailor  Armament  SEC  Policies  and  Procedures  for 

COTS  Implementation 


Translate 
SAP  terms 
in  the 
glossary 


\ 


Update 
CM  policy, 
including: 
versioning 
product 
delivery 


Armament  SEC  Organizational  Processes 

POLICIES  (All  Currently  Common) 


tC^2vII-Dev  vl.2  Common  Policies  Trace  Matm.xls 

Con-unon  Policy  Glossary  - 11  Mar  QS.doc 

Policy  AM-05  - 1 1  Mar  OS.doc 

PoUcv  CM-05  -  30  Dec  2&05.doc 

policy  DI-Q3  - 11  Mar  OS.doc 

PoUcv  OPM-C5  - 1 1  Mar  OS.doc 

PoUcv  PA-06  - 1 1  Mar  OS.doc 

PoUcvPE-05-llMar0S.doc 

PoUcv  ?FM-Q-t  - 1 1  Mar  OS.doc 

PoUc%-PM-Q6  - 11  Mar  OS.doc 

PoUcvR.M-05- 11  Mar08.doc 

PoUcv  SR-05  - 11  Mar  OS.doc 


CMMI-Dev  vl.2  Common  PoUcies  Trace  Matrix 
Common  PoUcy  Glossarx',  11  Mar  OS 
PoUcy  AM-05,  Acquisition  Management 
PoUcy  CM-05,  Configuration  Management 
PoUcy  DI-05,  Development  and  Integration 
PoUcy  OPM-05,  Organizational  Process  Management 
PoUcy  PA-Od,  Process  Assurance 
PoUcy  PE-05,  Product  Evaluation 
PoUcy  PFM-04,  Performance  Management 
PoUcy  PM-Od,  Project  Management 
PoUcy  RM-05,  Requirements  Management 
PoUcv  SR-05,  Status  Rexiew 


Approved 

Approved 

Approved 

Approved 

Approved 

Approved 

Approved 

Approved 

Approved 

Approved 

Approved 

Approved 


Policy  SST-01  -  25  Sep  09.doc 


! 


Policy  SST-01,  Service  System  Transition 


Add  new  policy  for 
Change  Mgmt 


Add  new  procedure 
for  Change  Mgmt 


All  CM  procedures 
tailored  ^ 


Procedures: 

Organizational  Level 

File  Name 

Title 

Status  1 

CPOOl  Organiz  Process  Mgmt  - 11  Mar  2008.doc 

CPOOl  Organiz  Process  Mgmt  -  llMar200S 

Approved 

CP002  PoUcv  Dex-  - 11  Mar  OS.doc 

CP002  PoUcy  Development  - 1 1  Mar  OS 

Approved 

CP003  Procedxire  Dev  - 1 1  Mar  OS.doc 

CP003  Procedure  Development  - 11  Mar  OS 

Approved 

CP004  Lifecvcle  Model  Dev  - 11  Mar  OS.doc 

CP004  Lifecycle  Model  Development  - 1 1  Mar  OS 

Approved 

CP005  Lessons  Learned  - 11  Mar  OS.doc 

CP005  Lessons  Learned  -  11  Mar  OS 

Approved 

CP006  Organiz  &  Project  Training  - 1 1  Mar  OS.doc 

CPOOO  Org  &  Project  Training  -  1 1  Mar  OS 

Approved 

CPOO"  Acquisition  Management  - 11  Mar  OS.doc 

CPOO^  Acquisition  Management  - 1 1  Mar  08 

Approved 

CP  CM002  Change  Requests  Problem  Rots  -  30  Dec  05.doc 

CP-CM002,  Change  Requests  Problem  Rpts 

Approved 

CP-CM003  Processing  BaseUne  Changes  -0"'  April  09.doc 

CP-CM003  Processing  BaseUne  Changes  -O'  April  09 

Approved 

CP-CMCC-  Creating  &  Releasing  Products  -  0"  April  C5.doc 

CP-CM004  Creating  &  Releasing  Products  -  O'  April  09 

Approved 

CP-CM006  Conducting  FCA  PCA  -  0"  April  09.doc 

CP-CM006  Conducting  FCA  PCA  -  O'  April  09 

Approved 

Approved 

CP-PACCl  Auditing  Projects  P,.eoorting  P,.esults  -  11  Mar 

CP-PAOOl  Auditing  Projects  Reporting  Results  - 11  Mar 

OS.doc 

OS 

CP-CMCC  1  Managing  Sotb-vare  CCB  Actix’ities  -  C"  April 

Managing  Sofhvare  CCB  Actixities  -  0"^  April  09 

Approved 

09.doc 

CP008  Service  System  Transition  -  25  Sep  2009.doc  CP008  Service  System  Transition  -  25  Sep 
2009.doc 


The  Army  ERP  COE  is  bridging  the  gap 


Engineering 


Business 


Backups 


Overarching  Project  Plan  Approach 


Overarching 
Army  ERP 
COE  Project 
Plan 


Streamlined  common 
processes 


Project  #  1 


Deltas  to  the  overarching 
Project  Plan 


•Streamlined  processes  followed  for 
every  project  iteration 
•More  efficient  use  of  Project 
Members'  Time 

•Simpler  to  review  and  manage 
•Less  chance  of  error  and  missed 
sections 


This  is  a  tried-and-true  approach  followed  by  multiple  current  Armament  SEC 

projects 


CMMI  for  Acquisition  (CMMI-ACQ) 

Considerations 


Acquisition  of  the  core  COTS  product 


•Continuous,  indefinite  partnership  with  software  provider  (SAP) 
•Software  development  of  the  base  product  is  managed  by  COTS 
company 

•Insight  into  SAP  software  development  practices  is  limited 


Acquisition  of  IT  services  specific  to  COTS  development  (SAP  skills) 

•Efforts  towards  an  Organic  base  of  skills  start  with  a  higher  percentage  of 
contractors 

•Human  Capital  Plan  goal  is  70%  government  and  30%  contractor  COE 
resources  thru  hiring  and  conversion 

•Best  Practices  from  CMMI-ACQ  will  be  referenced.  These  include  managing 
supplier  agreements,  verifying  and  validating  delivered  solutions. 


Configuration  Management  Tailoring: 

Versioning 


Traditional  Custom 
Development  Versioning 


COTS  Versioning 


SAP  ECC  v5.0 


Production  ERP 
environment 


Thursday  update 
Thursday  update 


Constant 

production 

data 

updates 


Weekly  online 
backup  process 


Version  numbers  are  assigned  by  the  vendor  to  the  out-of-the-box  COTS  product 


Configuration  Management  Tailoring: 

Product  Delivery 


> 


Traditional  Software 
Delivery 


COTS  Software  Delivery 


Development 

Environment 


CUSTOMER 


•Seamless  updates  to 
the  user 

•Training  provided  for 
user-impacting 


Packaged  software  delivered 
"fielded"  to  customer. 
Installation  required. 


The  ERP  organization  owns  the  development  as  well  as  production  environments 


Lessons  learned  while  tailoring 


^Translation  guide  of  SAP  terms,  roles  and  activities  helped  us  all  speak  the  same  'language' 
^Crosswalk  of  available  CMMI  models  showed  us  that  although  this  is  a  service-provider  system  with 
some  acquisition  pieces,  it  is  also  a  complex  development  effort  that  will  benefit  most  from  CMMI-DEV 
[^Configuration  Management  practices  -  versioning,  delivery  of  the  product  and  the  communication  to 
the  customer  -  follow  a  completely  different  path  than  most  ASEC  projects.  We've  tailored  the  policy 
and  procedures  accordingly. 

0Large  system  integration  efforts  such  as  an  ERP  have  huge  user  impacts.  The  Service  System 
Transition  (Change  Management)  process  area  from  CMMI-SVC  addresses  these. 

0COE  and  ASEC  have  a  mutually  beneficial  relationship:  COE  utilizes  CMMI-Level  5  developed  practices 

and  ASEC  expands  their  policies,  procedures  and  examples  to  include  COTS  systems 

0An  overarching  Project  Plan  approach,  with  smaller  plans  for  each  project  iteration,  suits  the  COE  ERP 

effort  best.  It  embeds  streamlined  processes  that  line  up  with  the  goal  of  integration  of  multiple 

systems 

0We  see  a  great  opportunity  for  use  of  CMMI  in  a  COTS  product  and  will  share  our  best  practices  with 
theSEI. 

0A  big  hurdle  in  our  understanding  of  COTS  development  efforts  is  that  a  developer  "configures"  the 
software  as  opposed  to  traditional  "coding"  in  software  development 


National 

Defense 

Industrial 

Association 


“The  Premier  Defense  Association!” 
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NDIA  Heritage 


*  1919  -  Founded  as  Army  Ordnance  Association  - 
AOA 

*  1948  -  Renamed  American  Ordnance  Association  - 
AOA 

*  1973  -  Renamed  American  Defense  Preparedness 
Association  -  ADPA 

*  1997  -  NDiA  created  from  merger  of  ADPA  and 
Nationai  Security  industriai  Association  (NSiA) 


MDIK 

National  Defense  Industrial  Association 


Vision 

America’s  leading  Defense  Industry  association 
promoting  National  Security 


Mission 

•  ADVOCATE:  Cutting-edge  technology  and  superior  weapons, 
equipment,  training,  and  support  for  the  War-Fighter  and  First 
Responder 

•  PROMOTE:  A  vigorous,  responsive.  Government  -  Industry 
National  Security  Team 


PROVIDE:  A  forum  for  exchange  of  information  between 
Industry  and  Government  on  National  Security  issues 


MDIK 

National  Defense  Industrial  Association 


About  Us 


*2jT7IQ9 


Non-profit,  educational  association 

Work  with  industry,  government  and  all  military 
Services 

1,565  corporate  members 

Over  73,600  individual  members  (22,000  Gov’t) 

52  Chapters 

32  Divisions 


MDIK 

National  Defense  Industrial  Association 


Activities 

•  Symposia  -  @  70-80  per  year  -  Policy,  Warfighting, 
Logistics,  Technical,  Systems  Acquisition,  International 
Cooperation,  Small  Business,  Homeland  Security,  etc. 
focus  areas 

•  Exhibitions  -  @  30  per  year  -  the  latest  technology  and 
defense  related  capabilities  on  display 

•  Advocacy  in  Washington  on  broad  industrial  base 
issues 

•  News  -  timely  views  from  the  Pentagon,  the 
Administration,  Congress  and  Industry  via  National 
DEFENSE  magazine 


Studies,  reports,  assessments,  reviews  for  government 
entities 


MDIK 

National  Defense  Industrial  Association 


Chapters 


*  Extend  NDIA  ‘reach’  across  the  U.S. 

*  Provide  geographic  focus  for  NDIA  vision  and 
mission 

*  Governed  with  By-Laws 

*  Conferences,  social  activities,  award  &  scholarship 
programs,  etc. 


MDIK 

National  Defense  Industrial  Association 


Divisions 


Provide  ‘functional’  focus  for  NDIA  mission 

Populated  by  corporate  members 

‘Mission  Area’  oriented 

Governed  by  formal  Charters 

Conferences,  studies,  workshops,  seminars, 
awards,  etc. 


Divisions  --  Technology 


Space 

Missile  Defense 
Armaments 

Science  &  Engineering  Tech 
Munitions  Technology 
Chemical  Biological  Defense 
Bombs  &  Warheads 
Robotics 


Ballistics 

Tactical  Wheeled  Vehicles 
C4ISR 

Systems  Engineering 
Technical  Information 
Test  &  Evaluation 
Homeland  Security 
Manufacturing 


Combat  Vehicles 


MDIK 

National  Defense  Industrial  Association 


Divisions  --  Warfare 

•  Expeditionary  Warfare 

•  Undersea  Warfare 

•  Strike,  Land  Attack  &  Air  Defense 

•  Air  Targets,  UAVs  &  System  Ranges 

•  Combat  Survivabiiity 

•  Speciai  Operations/Low  Intensity  Conflict 


MDIK 

National  Defense  Industrial  Association 


Divisions 


*  Environment  &  Energy 

*  International 

*  Health  Affairs 

*  Logistics 


STEM  Workforce 


Poiicy 


Legislative  Information 
Procurement 
Government  Policy 
Small  Business 


MDIK 

National  Defense  Industrial  Association 


Industrial  Committees 


Committee 

Sponsor 

Ammunition  Producers 

CG,  JMC  /  PEG  Ammo 

Test  &  Evaluation 

Dir,  Test  &  Eval,  OSD 

Small  Arms  Producers 

Vacant 

Chem-Bio  Defense  Acquisition 
Initiative  Forum 

JPEO,  Chem-Bio 

Program  Management 

AT&L  (A&T) 

Biometrics 


Dir,  DoD  BTF 


MDIK 

National  Defense  Industrial  Association 


Affiliates 


Association  for  Enterprise  Information 


nisa 


National  Training  &  Simulation  Association 


PRECISION  STRIKE 
ASSOCIATION 


Precision  Strike  Association 


Women  In  Defense 


National 

Defense 

Industrial 

Association 


“Your  Premier  Defense  Association!” 
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Making  the  CMMI®  Sing 

A  Framework  for  Performance  Excellence 

CMMI®  Technology  Conference  and  User  Group 

November  17^^,  2009 


Jeffrey  L  Dutton 
Chie  f  Engineer 
Jacobs  Technoiogy  Inc.  ITSS 


This  presentation  spans 
TWO  sessions 


2009  CMMI  Technology  Conference  and  User  Group 


Slide  No.  2 


JACOBS 


Administrivia 


•  Trademarks  and  Service  marks 

—  ITIL®  is  a  Registered  Trade  Mark,  and  a  Registered  Community  Trade  Mark  of  the 
Office  of  Government  Commerce,  and  is  Registered  in  the  U.S.  Patent  and 
Trademark  Office 

—  IT  Infrastructure  Library®  is  a  Registered  Trademark  of  the  Central  Computer  and 
Telecommunications  Agency  which  is  now  part  of  the  Office  of  Government 
Commerce 

-  ®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon 
University 

—  SM  SCAMPI  is  a  Service  mark  of  Carnegie  Mellon  University 

•  Who  I  am: 

-  Chief  Engineer,  Jacobs  Technology,  Inc./ITSS 

-  SCAMPI  Lead  Appraiser 

-  (Lean)  Six  Sigma  Black  Belt 

-  Certified  Scrum  Master 

-  Member,  NDIA  Systems  Engr  Steering  Committee 

-  Member,  NDIA  CMMI  Working  Group 

-  Member,  CMMI-SVC  Advisory  Group 
—  Visiting  Scientist,  SEI 


2009  CMMI  Technology  Conference  and  User  Group 
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Outline 


•  Goals  of  a  performance  improvement  approach 

•  Discovering  some  driving  principles 

•  Attributes  of  some  performance  improvement 
approaches 

•  Our  journey 

•  Introducing  the  Framework  for  Performance 
Excellence 

•  Value  propositions  of  framework  components 

•  Making  the  Framework  sing 


2009  CMMI  Technology  Conference  and  User  Group 
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Goals  of  a  Performance  Improvement  Approach 


•  Respond  to  business  objectives  or  solve  problems 

•  Exhibit  positive  return  on  investment 

•  Produce  sustainable  improvements 

•  Be  transferrable  across  projects  and 
organizations 

•  Produce  results  fast  enough  to  make  business 
sense 


2009  CMMI  Technology  Conference  and  User  Group 
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Outline 


•  Goals  of  performance  improvement 

•  Discovering  some  driving  principles 

•  Attributes  of  some  performance  improvement 
approaches 

•  Our  journey 

•  Introducing  the  Framework  for  Performance 
Excellence 

•  Value  propositions  of  framework  components 

•  Making  the  Framework  sing 


2009  CMMI  Technology  Conference  and  User  Group 
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Some  Driving  Principies 


•  Focus  on  performance  and  quality  objectives 

•  Direct  involvement  of  leadership 

•  Process  ownership 

•  Improvement  velocity 


2009  CMMI  Technology  Conference  and  User  Group 
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Focus  on  Performance/Quality  Objectives 


•  Examples  of  performance  objectives 

-  Reduce  software  life  cycle  time  frame 

-  Increase  level  of  service 

-  Respond  to  changes  in  customer  demand  in  three 
months  or  less 

-  Reduce  cost  of  development  by  35% 

•  Examples  of  quality  objectives 

-  Meet  service  levels  99.9%  of  the  time 

-  Reduce  delivered  defects  to  less  than  3  per 
1 ,000,000  opportunities 


2009  CMMI  Technology  Conference  and  User  Group 
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Why  are  Performance/Quality  Objectives  Important? 


Because  they  change 


•  The  improvement  approaches  chosen 

•  Interpretation  of  CM  Ml  practices 

•  Workflow  measures  in  Value  Stream  Mapping 

•  Measurement  objectives 

•  Which  CMMI  Process  Areas  to  implement 

•  What  Maturity  or  Capability  Levels  to  target 

•  What  part  of  the  organization  to  improve 

•  How  much  you’re  willing  to  invest 


2009  CMMI  Technology  Conference  and  User  Group 
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Cost 

Value 


Estimated  ROI 


Return  on  Investment  Envelope 


MAXMUM  ROI  CURVE 


Sharp  focus  on  organizational 
performance  and  quality  goals 


WORST  CASE 
VE 


Break-Even  Line 


Little  or  no  focus  on  organizational 
performance  and  quality  goals 


.evel  2 


Level  3 


Level  4 


Level  5 


Focus  on  Organizational  Performance 

and  Quality  Goals 


2009  CMMI  Technology  Conference  and  User  Group 
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Direct  Leadership  Invoivement 


•  “Allowing”  the  organization  to  improve  is  often  not  enough 

-  Resources,  personnel,  money 

—  Some  level  of  process/work  product  review 
—  Support  for  organizational  change 

-  Approval  and  support  of  process  changes 

•  Direct,  active  involvement  is  key 

—  Tie  effort  to  real  business  objectives  and  issues 
—  Be  demanding  of  results  in  a  meaningful  time  frame 
—  Set  high  level  performance  and  quality  goals 

-  Get  “heroes”  and  key  personnel  directly  and  personally 
involved 


2009  CMMI  Technology  Conference  and  User  Group 
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Process  Ownership 


•  Levels  of  removal  from  process  ownership 

—  Hire  a  professional  to  come  in  and  write  your 
processes  (increasingly  rare) 

-  Form  an  SEPG  of  “process  people” 

•  Buy-in  strategies 

•  Dealing  with  “heroes” 

•  Mandates  for  use  of  processes  (!) 

•  Ownership  by  process  “doers” 

-  Charge  the  “heroes”  with  leading  performance 
improvement 

—  Exactly  as  intended  by  Lean  Thinking 
—  Make  performance  improvement  everyone’s  job 

JACOBS 
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Improvement  Velocity 


•  Velocity  =  speed  in  a  specific  direction 

•  Improvement  “at  the  speed  of  business”  is  the  key 

•  Barriers  to  high  velocity: 

-  Lack  of  focus  (objectives,  issues,  scope,  etc.) 

-  Lack  of  leadership 

-  Processes  not  owned  by  “doers” 

-  Low  process  maturity 

-  Misunderstanding  of  CMMI  and  other  approaches 
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Outline 


•  Goals  of  performance  improvement 

•  Discovering  some  driving  principles 

•  Attributes  of  some  performance  improvement 
approaches 

•  Our  journey 

•  Introducing  the  Framework  for  Performance 
Excellence 

•  Value  propositions  of  framework  components 

•  Making  the  Framework  sing 
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Attributes  of  Performance  Improvement  Approaches 


•  Lean  Thinking: 

—  Pros: 

-  Cons: 

•  TheCMMi: 

—  Pros: 

-  Cons: 

•  The  information  Technoiogy  infrastructure  Library: 

—  Pros: 

-  Cons: 

•  Six  Sigma: 

—  Pros: 

-  Cons: 
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Our  Lean/CMMI/ITIL/Six  Sigma  Journey 


Customer  Efforts  Internal  Efforts  Discoveries 


CMMI-Software  ML2 
Lean  CMMI-DEV  ML2 


CMMI-Software  ML3 


Lean  CMMI-DEV  MLS 


Lean  CMMI-DEV  MLS 


Lean  CMMI-DEV  HM 


Lean/ITIL  CMMI-SVC  (C) 


Lean  CMMI-SVC 
Lean  CMMI-SVC 


Lean/CMMI  work  together 
Lean/CMMI  offers  reduced 
costs  and  schedule 
1®*  High  Veiocity  effort 
Vaiue  of  Performance  Goals 
Directly  involved  leadership 
integration  of  CMMi  &  6a 
Lean/iTiL/CMMi-SVC 
offers  high  vaiue 
Continuous  CMMi-SVC 
PAs  in  iSO  certified  org. 
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Outline 


•  Goals  of  performance  improvement 

•  Discovering  some  driving  principles 

•  Attributes  of  some  performance  improvement 
approaches 

•  Our  journey 

•  Introducing  the  Framework  for  Performance 
Excellence 

•  Value  propositions  of  framework  components 

•  Making  the  Framework  sing 
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Lean/CMMI/SixCJ  Venn  Diagram 
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The  Framework  for  Performance  Excellence 


Legend: 

Driving 

^  \ 

Principles 

linprovsmiint  ( 

) 

Approaches  ^ 

n _ y 
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Outline 


•  Goals  of  performance  improvement 

•  Discovering  some  driving  principles 

•  Attributes  of  some  performance  improvement 
approaches 

•  Our  journey 

•  Introducing  the  Framework  for  Performance 
Excellence 

•  Value  propositions  of  framework  components 

•  Making  the  Framework  sing 
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Value  Propositions  for  Framework  Components 


•  The  CMMI 

•  Lean  Thinking 

•  Six  Sigma 

•  iTiL 
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Capability  Maturity  Model  Integration 


•  What  is? 

-  Models  (goals,  practices,  informative  material) 

-  SCAMPI  appraisal  methods 

-  Core  training  (SEI  authorized) 

•  Value  proposition: 

-  Domain-specific  best  practices  (development,  services,  and  acquisition) 

-  Practices  for  improvement  infrastructure 

-  Framework  for  continuous  improvement 

•  Maturity  Levels 

•  Process  Area  Capability  Levels 

-  Robust,  extensible  appraisal  methods 

•  Course  correction 

•  Learning  mechanism 

•  Benchmarking 

•  Downside: 

-  No  improvement  approach  or  strategy 

-  Needs  focus  and  leaning 

•  Integration  with  other  approaches: 

-  Synergistic  with  Lean 

-  Actualizes  Six  Sigma 

-  Implements  ITIL 
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Lean  Thinking 


•  What  is? 

-  Focus  on  customer  value 

-  Value  stream  mapping  (workflows) 

-  Cadence  and  synchronization 

-  Organizational  rapid  learning 

-  Process  doers  are  process  owners 

-  Reliance  on  tacit  knowledge  and  skilled  team  members 

-  Agile  project  management 

•  Value  proposition: 

-  High  velocity  (Presentation  Wednesday  SAM) 

-  Lean  (smart)  processes  and  process  efficiency 

-  Builds  mature  teams  quickly 

-  Rapid  response  to  customer  pressures 

•  Downside: 

-  No  improvement  infrastructure 

-  Suffers  from  lack  of  consistency  and  persistence 

•  Integration  with  other  approaches: 

-  Synergistic  with  CMMI  models 

-  Leverages  Six  Sigma 

-  Sharpens  business  context  of  ITIL 
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Lean  Value  (ROI) 


Degradation  Curve  for  the  Lean/Agile  Value  Proposition 


Family  of  Lean/Agile  Constructs 
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Six  Sigma 


•  What  is? 

-  Statistical  mechanisms  for  process  control 

•  Process  variability 

•  Central  tendency 

-  Some  mechanisms: 

•  Regression  and  correlation 

•  Tests  of  Hypothesis 

•  Analysis  of  variance 

•  Statistical  process  control 

•  Experimental  design 

•  Process  performance  modeling  and  optimization 

•  Value  proposition: 

-  Allows  prediction  of  project  performance 

-  Leading  vs.  lagging  indicators 

-  High  degree  of  process  control  (e.g.  six  sigma) 

•  Downside: 

-  High  cost 

-  Extensive  timelines  (improved  by  lean) 

•  Integration  with  other  approaches: 

-  Fully  integrable  with  CMMI 

-  Energized  by  lean  (shorter  cycles/more  data)  JACOBS 
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Process  Stability 


Sample  Process  Control  Chart 


Upper  Specification  Limit  -  USL  (Voice  of  the  Customer) 


Lower  Control  Limit  -  LCL  (Naturai  Process  Limit) 


Lower  Specification  Limit  -  LSL 

Time  ^ 
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Process  Capability 


Information  Technology  Infrastructure  Library 


•  What  is? 

-  Best  practices  for  IT  service  operations 

-  Fair  implementation  guidance 

-  ITSM  life  cycle 

•  (Strategy/Design/Transition/Operation/Continuous  Improvement) 

•  Value  proposition: 

-  Excellent  set  of  IT-  specific  practices 

-  Several  useable  ITSM  processes 

-  Personal  knowledge  certifications 

-  ISO  20000  registration 

-  Some  guidance  for  setting  objectives  and  strategy 

•  Downside: 

-  Little  support  for  “organization  for  improvement” 

-  No  framework  for  benchmarking  performance  improvements 

•  Integration  with  other  approaches: 

-  Works  well  with  CMMI-SVC 

-  Can  be  benchmarked  with  CMMI  SCAMPI  A  (presentation  Wednesday  10AM) 

-  Orthogonal  to  Six  Sigma 

-  Organizational  context  improved  with  Lean  Thinking 
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Relative  Contributions  Fully  Integrated  Framework 

(CMMI-SVC  Example) 


■  G  Sigma 

■  Lean 

■  ISO  20000 


Vf 

3 

O 
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V 
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Speed 

mprovement 

Framework 

u  St  Appra  isal 
Method 

E 

'c 
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01 

■  CMMI-SVC 

■  ITIL 
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Outline 


•  Goals  of  performance  improvement 

•  Discovering  some  driving  principles 

•  Attributes  of  some  performance  improvement 
approaches 

•  Our  journey 

•  Introducing  the  Framework  for  Performance 
Excellence 

•  Value  propositions  of  framework  components 

•  Making  the  Framework  sing 
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Making  the  Framework  Sing 


•  Applying  the  Framework 

•  Driving  principles 

-  Focus  on  performance  and  quality  objectives 

-  Direct  involvement  of  leadership 

-  Process  ownership 

—  Improvement  velocity 

•  Choosing  the  improvement  approaches 

•  Tuning  the  Framework  -  some  examples 
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Applying  the  Framework 
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Example  1 :  Small  SW  Development  Organization 


•  Performance  Objectives: 

-  Negotiated  schedules  are  estimated  and  met 
with  no  more  than  a  10%  variance 

—  Financial  costs  within  a  10%  variance 

-  Customer  survey  scores  of  90+%  satisfaction 

—  Delivered  product  and  development  iterations 
meet  or  exceed  committed  requirements  100% 
of  instances 

—  Customer  sign  off  occurring  within  1  week  after 
project  completion 

•11  Months  to  achieve  goals  and  ML3 
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Performance  Objectives: 

-  Negotiated  schedules  are  estimated  and  met  with  no  more  than  a  10%  variance 

-  Financial  costs  within  a  1 0%  variance 

-  Customer  survey  scores  of  90+%  satisfaction 

-  Delivered  product  and  development  iterations  meet  or  exceed  committed  requirements  100%  of  instances 

-  Customer  sign  off  occurring  within  1  week  after  project  completion 


nization 


FOCUS  ON 
BUSINESS 
CONTEXT 


Lean  for  Low 
Defects  and 
Improvement 
Velocity 


LEADERSHIP 
AND  PROCESS 
OWNERSHIP 


Sr.  Leadership 

Defined  objectives  and 
participated  directly  in  process 
discussions 


One  week 
improvement 
cycles 


CMMI-DEV 


INTEGRATED 

CMMI-GASED 

FRAMEWORKS 


Results:  Achieved  all 
performance  goals  and  CMMI 
MLS  in  11  months 


H4J|uroiiiLrit5a 
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Example  2:  IT  Service  Organization 


•  Large  IT  commercial  organization 

•  Internal  (Lean)  CMMI-DEV  MLS  software  dev. 
organization 

•  Performance  Objectives: 

-  Mistake-free  processes  and  services 

-  Seamless  flow  between  business  departments 

—  Single  ownership  of  services 

—  Delivered  services  meet  or  exceed  Service  Level 
Agreement  (SLA)  100%  of  instances 

-  Develop  credible  proof  of  delivery  capability  and 
continuous  improvement 
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Example 


Performance  Objectives: 

-  Mistake-free  processes  and  services 

-  Seamless  flo\A/  between  business  departments 

-  Continuous  improvement  of  defined  services 

-  Single  entry  of  information/data 

-  Single  ownership  of  services 

-  Delivered  services  meet  or  exceed  Service  Level  Agreement  (SLA)  1 00%  of  instances 

-  Develop  credible  proof  of  delivery  capability  and  continuous  improvement 


)rganization 


Lean  VSMs  for 
Customer  Focus 
and  Waste 
Elimination 


LEADERSHIP 
AND  PROCESS 
OWNERSHIP 


Sr.  Leadership 

Defined  objectives  and 
participates  directly  in  process 
discussions 


FOCUS  ON 
BUSINESS 
CONTEXT 


CMMI-SVC 


INTEGRATED 

CMMI-GASED 

FRAMEWORKS 


Results  to  date:  Benchmark 
Kaizens  complete.  Services 


13  Month  Goal  to 
achieve 
objectives  and 
ML3 


1 


defined.  Initial  process 
interfaces  reconciled. 

JACOBS 
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•  To  discuss  further,  contact  me  at: 


QUESTIONS? 
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Process-Performance  Based 
Reliability  (PPBR) 


Standardization  of  Organizational  Data  Analysis 

via 

CMMI-Causal  Analysis  and  Resolution  (CAR) 


November  17-19, 2009 


William  B.  Winkel 


Agenda 

•  Why  is  a  new  process  for  reliability  prediction  needed 

•  How  can  a  process  be  developed  around  CMMl-CAR 

•  What  issues  must  the  new  process  address  relative  to 
the  organization’s  process  health 

•  Summarize  the  process  and  provide  one  sample 
calculation 


Industry  Trends  are  DrMngjthe  Need  for  New  Rejj 
T>e^igrrand  Analysis  Methods 


•  Contractors  must  “build  the  case”  for  improving  product  reliability  during 
product  development  cycle 

•  Ernest  Seglie,  Christopher  Dipetto,  Office  of  the  Secretary  of  Defense,  ''Report  of  the  Reliability  Improvement  Working 
Group”,  September  4,  2008^^^ 

•  Ministry  of  Defence  Standard  00-42,  Reliability  and  Maintainability  (R&M)  Assurance  Guidance  Part  3  R&M  Case, 
Issue  2  Publication  6  June  2003 

•  SAE JAiooo-i,  "Reliability  Program  Standard  Implementation  Guide”,  1999-03-01 


•  Contractors  will  execute  pay-for-performance  contracts  (PBL) 

•DODD  ^000.1,  Department  of  Defense  Directive,  "The  Defense  Acquisition  System”,  May  12,  2003,  paragraph  Ei.i.iy 


•  Organizations  must  demonstrate  continual  process  improvement  via  process 
performance  models 

•CMMI®  for  Acquisition,  Version  1.2,  CMU/SEI-2007-TR-017,  ESC-TR-2007-017,  November  2007  [5I 
•CMMI*  for  Development,  Version  1.2,  CMU/SEI-2006-TR-008,  ESC-TR-2006-008,  August  2006  [^1 


•Current  Reliability  prediction  methods  have  deficiencies 


Literature  review  has  identified  new  direction  for  Reliability  Engineering 
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•  Left  unmanaged,  organizations  have  limited  visibility  of  reliability  and  cost  growth 
between  phases  of  development 

•  Normalized  defect  counts  are  unavailable  for  between-phase  comparisons 

•  Defects  are  not  uniformly  categorized  between  development  activities 

•  Corrective  action  effectiveness  is  unknown 

•  Unincorporated  corrective  action  varies  randomly  from  last  phase  performance 

•  A  standard  process  and  single  web-based  tool  provides  synergy  across  multiple 
functional  groups  within  an  organization 

•  Normalized  defects  are  continuously  monitored  and  measured  within  and  between  phases 

•  Correlations  are  established  between  categories  of  development  and  field  defects 


jj  asDD 

Individual  Process  Categories  Between  Two  Phases  of  Development 
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nswer  Five  Critical 


ions 


•  Is  the  probability  of  a  field  defect  warrant  the  cost  of  determining  and 
incorporating  corrective  action? 

•  Are  defects  falling  through  the  cracks? 

•  Are  the  separate  FRB  s  within  the  organization  performing 
satisfactorily? 


•  Is  the  correction  capability  of  each  organizational  sub-process 
maintaining  control? 


Has  reliability  growth  occurred? 


PPBR  Introduces  a  CMMI-CAR  Compliant  Closed-Looj 
^^^Cgcpeeth/e  Action 


•  Reliability  activities  are  integrated  systematically  across  an  organization 

•  Measurement  performance  of  analyst,  Failure  Review  Board,  sub-process,  and 
organizational  management 

•  4-step  process:  Product  reliability  is  not  simply  measured  it  is  managed  (via  business 
decisions)  to  ensure  growth  between  phases  of  program  development 


ID  (PID) 


SUPPLIER 


SWp^iintg 


Euppllir  Error 


Suppiry  Chain  Central 


fVnduct  IliieejJrty 


jHlppIng 


if’iJC  iVl  out 


Organization 

I 


Prevention 


Product  Design 


MflCtifinital  Der^ian 


Mochafilcal  Anafysi^. 


Proanti  liiiayniy 


Data  Enyirworing 


CMflitJl  Systems 


RF  Foww  Systftmt 


Syslemfl  tnrjinwHrnrf 


Sottware 


Test  ErijjinB&nnj 


Lliifiiii'eTiiig 


Product 

Manufacture 


Customer  Monitor 


Sflfl  Alliaclied  F|(o 


SyEitemEi  Engingifnng 


Recoww  Eccrter  s 


Detection 

I 


Mis^^unAsturance 


Lcgj&tic^i  Engineertng 


Product  TeitAr^g  1 


Product  TosiAroa  N 


0|Mf^iciri£Pepair 


CUSTOMER 


Resolution 

^ 


Mi^'fiuri  A^^urijnew 


Cuslomor  Error 


System  SupportabitiEy 


Not  all  defects  are  within  the  span  of  control  of  the  organization 


Organizational  processes  are  categorized  as  related  to  prevention,  detection, 
and  resolution 


PID  defines  the  sub-process  that  the  defect  has  escaped  from 


•  Make  decision  to  determine  root 
cause  for  current  PID 

•  Assign  PID  to  a  new  or  existing  docket 

•  Complete  the  path  associating 
process  to  physical  defect 

•  Defect  Description  (DDST)  and  Failure 
Mode  ID  (FMID)  are  docket  level  attributes 

Examples: 

Failure  Description:  Solder  joint  is  cracked  on  PLCC, 
MC68HC11F1FN. 

PID  Structurel:  Prevention/Product  Design/Mechanical 
Analysis/PoF/lnsufficient  solder  height 

PID  Structure  2:  Prevention/Product 

Manufacture/Assembly,  Manual/Broken  or  damaged 
components 


Process 

Identification 

Name: 

Location: 

IVIechianical  Design 

IVIechanical 

Analysis 

Product  li 

Sub  process 
Identification 

Name: 

Modules 

Wfired 

Chassis 

System 

Installation 

PoF 

'Vibration 

Thermal 

Com  po  n  ents 
Engineering 

Materials  Si 

P  roc  esses 

Defect  Description 
(DOST) 

do  1 1  eci;  C  usto  m  e  r  Re  q  u  i  re  m  e  nl^s 

AllO'Cate  Requirements  Internally 

Allcscate  Requirements  Externally 

Select  Pa rts/Mate rials 

Perform  Analysis 

Generate  Soh e matic/D ra wi ng 

Generate  Test  Plans 

PRODUCT  MANUFACTURING 


Location  2 


Location  3 


Location  4 


Location  5 


BoldText:Sub 

procsssesfor 


/ 


Adhesive  Dispense 

/Idjfiesire  foo  /ligi? 
Mssmsc/fnedeqaafe  Bdheme^ 

At  connectors 
On  cover  insulators 
Adiiesi^e 

Expired  siieff  life /nafe/iaf 
Assembly,  Manual 
Sreken/damaped  componenfs  * 
Co/nponenfs  in  mng  locdfms 
Power  transistors 
Passive  components 
Gofnponenf  onenfafion 
Power  transistor 
Passive  components^^''"""^ 
Improper  torque  applied 
/mpreper  hardmre  i/isfaiiafion 
Mssed  assereWy  sfep 
\  t/nif  dareaped  dump  assere  Wy 
Bonding 

Expired  siieif  life  reafenai 
/mprepedy  maciimed  prefareis 


Heading:  Manufacturing  Cell  Locations 


Italics:  Defect  Descriptions  (DDST) 


G  reen  Font:  Failure  Mode  IDs.  FMlDs  are 
not  standardized  since  there  are  an  infinite 
list  ofthenn.  They  are  used  by  the  FRACAS 
tool  to  track  incidents  that  have  a  common 
process  ID  and  Defect  descriptions. 
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step  3^  AssjgJ3Xoi:i:e£tive  Actio^racking  Index 


•  Corrective  action  resides  in  one  of  two 
states  “incorporated”  or  “not 
incorporated” 

•  Defects  reside  in  one  of  two  states 
“customer”  or  “non-customer”  returns 


4  key  docket-level  parameters  provide 
state-control 

•  Program  Phase,  Corrective  action  Index, 
Fail  Date,  and  Customer  return  status 


Phase  1 


Phase  2 


Phase  ? 


Phase  4 


'  Correct/Ve  Act/on  iVot  kofpofflterf  ► 

Specific  CA  Path 

I 


I 


Pro|rarnPh4  =  0, 

CAI  Stats  =  NQne, 
E^pa:tsdlntl6ntCiDunt  =  N0 

I 


> 


Mofiitor  FRB/CAB 


Pm|rarnPh^  =  l, 

CAI  Stats  =  ^ 

E^psdsd  IrKkjsnt  Caufit^^ 


Moflitor  FRB/CAB 


Prairarn  Phass  =  2, 

CAI  Stats  =  None, 

Eipsttsd  Incident  Count:  NO 


V  Monitor  FRB/CAB 


[0] 


[il 


Min{  Incident  FailE'ate] 


[2]  A  [3] 


Incident  Fail  ['ate 


[3} 


[Index] 

[Time] 


Correct/Ve  Actio/i  korporaterf 


Program  Phase 


Program  P  has  =  0, 

CAI  Stats  =  CAI-1, 
E3!psctedlncidentC0unt=Nl 


SDD 


MoiitorCA 


Program  Phsa  =  l, 

CAI  State  =  CAI-1, 
&pstiid]jj[kjent  Count  =  N1 

I 


-h 


Pilot 


.  Monitor  W 


Program  Ph^  =  2, 

CAI  Stats  =  cll-l, 

Expected  Incident  Count  =  N1 


LRIP 


I 


State  sequencing  for  non¬ 
customer  defects  is  completely 
automated 


step  4:  The  Output  Metrics  Answer  the  5  Critical 


•  FRB  Effectiveness  example  answers  only  one  of 
5  key  questions 

•  Affects  only  dockets  that  have  not  had 
corrective  action  incorporated 

•  Is  only  meaningful  when  measured  across 
sub-processes  not  within  sub-processes 

•  Alarms  monitor  rate  of  defect 
accumulation 

•  Provides  three  measurements  of 
improvement 


F’logram  A  -  E'eUveen  Phase  C  efect  Status 


Parameter  DGfinitions: 

NDl  Normalized  defed  ratio  for  sub-process  1  during  the  baseline  phase 
ND2  Normalized  defed  ratio  for  sub-process  1  during  the  phase  subsequent  to  the  baseline 
N1  Total  number  of  defeds  associated  with  sub-process  I  colleded  at  the  end  of  the  baseline  phase 
N2  Total  number  of  defeds  associated  with  sub-process  Icolleded  at  the  end  of  the  subsequent  phase 
T1  total  number  of  assemblies  at  risk  (assembled)  during  the  baseline  phase 
11  total  number  of  assemblies  at  risk  (assembled)  during  the  subsequent  phase 
N  The  adual  number  of  defeds  currently  colleded  in  a  docket  for  the  current  phase 
T  The  adual  number  of  assemblies  currently  at  risk,  i.e.,  defined  as  Ok  for  stores 
Cpk-FRB  The  average  corredion  capability  of  FRBs  within  the  organization 
Sub-process  alarm  Indicates  that  the  rate  of  defeds  colleding  in  the  docket  will  cause  the  last  phase  limit  to  be  e>!ceeded 

Parameter  Calculations: 

NDl  =  Nl/Tl  For  the  baseline  phase 

NDi  =  Ni/Ti  In  general 

Sub-process  alarm  =  N  >(Nl/Tl)TCp^.fRg  Are  defeds  accumlating  ata  rate  higher  than  expeded? 
lO  The  alarm  is  ALWAYS  referenced  to  phase  1,  i.e,  (Nl/Tl) 


PID/DDST 

SDD 

LRIP 

FSP 

Baseline  Comparison 

Continued  Improvement 

SDD/LRIP 

SDD/FSP 

SOD/LRIP 

LRIP/FSP 

1 

0.95 

0.75 

0.65 

-0.21 

-0.32 

-0.21 

-0.13 

1 

1.2 

0.62 

0.6 

-0.48 

-0.50 

-0.48 

-0.03 

3 

0.35 

0.9 

0.45 

1.57 

0.29 

1.57 

-0.50 

4 

0 

0.6 

0.45 

-0.25 

Phase  to 

Average 

0.625 

0.7175 

0.5375 

Phase 

Std  Deviation 

0.548483 

0.138654 

0.103078 

Baseline 

Average 

0.29 

-0.18 

Connparison 

Std  Deviatiori 

1.12 

0.41 

Continued 

Average 

0.29 

-0.22 

Improvement 

Std  Deviation 

1.12 

0.25 

Results  can  Demonstrate  the  Effectiveness  of  FRB  and  Provide  a 
CMMN 


•  Metrics  show  negative  slopes  for 
decreasing  defect  count 

•  Single  phase  comparisons 
measure  absolute  performance 


V  =  -O.O^SVx  -I-  o.yi-42 


Absolute  Phase-to-Phase  Comparison 

Figure  1 

0.0 
0.7 
0.6 
0.6 
0.4 
0.3 
0.2 
0.1 
O 


-0.22273^4  0.7000 


Phase  Average 
I  Sta  n  d  a  r  d  □  e vi  atio  n 


Improvement  in  Process  Average 

Figure  2 


■  Baseline  Conipai  ison 
▲  Continued  Improveinent 

-  FRBPF'm 


Phase 


•  CMMl  Process  performance 
model  measures  consistent 
performance 


•  Standard  deviations  provide 
evidence  of  decreased  dispersion 
between  phases 


Improvement  in  Defect  Standard 
Deviation 

Figure  3 


▲  B  as  e  I  i  n  e  Co  m  p  a  r  iso  n 
♦  Continued  Improvement 
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Summary 

•  New  industry  requirements  require  a  fresh  look  at 
reliability  prediction 

•  CMMI-CAR  integrates  the  physical  and  process  aspects  of 
failure 

•  5  critical  questions  define  the  algorithm  for  corrective 
action  control  and  measurement 

•  CMMI-PPMs  are  developed  around  the  measured  results 
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Questions? 
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SCAMPI  Calendar  Year-End  2008  Statistics 


Cumulative  #s: 

CYE07 

CYE08 

Increase 

%  Increase 

#  Appraisals  Performed 

3,113 

4,134 

1,021 

32.8% 

#  Unique  Organizations  Appraised 

2,674 

3,446 

772 

28.9% 

#  Unique  Participating  Companies 

1,882 

2,544 

672 

35.7% 

#  Re-appraised  Organizations 

361 

564 

203 

56.2% 

#  Unique  Projects 

14,620 

21,141 

6,521 

44.6% 

CMMI  V1.3 

Software  Engineering  Institute  CarnegieMellon  October  6,  2009 

©2009  Carnegie  Meiion  University 


CMMI  Transition  Status 
Reported  to  the  SEi  as  of  10-31-09 


Training 

Introduction  to  CMMI  -  108,724 

Intermediate  CMMI  -  2,990 

Understanding  CMMI  High  Maturity  Practices  -  577 

Introduction  to  CMMI  VI  .2  Supplement  for  ACQ  -  1 ,050 

Introduction  to  CMMI  VI  .2  Supplement  for  SVC  (1  Day)  -  1 ,024 

Introduction  to  CMMI  Services  VI  .2  (3  Day)  -  102 

Authorized/Certified 

Introduction  to  CMMI  VI  .2  Instructors  (63  authorized)  -  388 

CMMI-ACQ  VI. 2  Instructors  (all  certified)  -  57 

CMMI-SVC  VI  .2  Instructors  (all  certified)  -  94 

SCAMPI  VI  .2  Lead  Appraisers  (all  certified)  -  466 

SCAMPI  VI  .2  B  &  C  Team  Leaders  (all  authorized)  -  531 

SCAMPI  VI  .2  High  Maturity  Lead  Appraisers  (all  certified)  -  144 

CMMI-ACQ  VI  .2  Lead  Appraisers  (all  certified)  -  48 

CMMI-SVC  VI  .2  Lead  Appraisers  (all  certified)  -  1 01 
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CMMI  V1.2  Foreign  Language  Translation  Status 
Reported  to  the  SEI  as  of  10-31-09 


XMMI-DEV  VT;2 

Language 

Status 

Japanese 

Completed  August  2007.  Intro  course  translated  October  2007 

Chinese  (Traditional) 

Completed  December  2007 

French 

Completed  August  2008 

German 

Completed  April  2009.  Intro  course  translated  October  2009 

Spanish 

Completed  in  June  2009 

Portuguese 

Underway,  to  be  completed  in  November  2009 

CMMI-ACQ  VI  .2 

Language 

Status 

Chinese  (Traditional) 

Completed  April  2009 

CMMI-SVC  V1.2 

Language 

Status 

Chinese  (Traditional) 

Underway,  to  be  completed  2009-2010 
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Introduction  to  CMMI  Attendees 
Cumulative  as  of  8-31-09 


120000 
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□  CMM 
Intro 
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□  CMMI 
Intro 
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Introduction  to  CMMI  Attendees 
Cumulative  as  of  10-31-09 


□  CMM  Intro 
(discon 'td. 
12/31/05) 


□  CMMI  Intro 
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Number  of  Appraisals  Conducted  by  Year 
Reported  as  of  10-31-09 


1200 
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□  SPA  ■CBAIPI(Discontinued  after  12/31/2005)  DSCAMPI  v.X  ClassA 
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CMMI  Adoption  Has  Been  Broad 


25  countries  with  10  or  more 
appraisals  (Aug  06  ->  Jul  08): 


USA 

598  ->  1034 

China 

158  ->  465 

India 

177  ->  323 

Japan 

155  ->  220 

France 

65  ->  112 

Korea  (ROK)  56 ->107 

Taiwan 

31  ->  88 

Brazil 

39  ->  79 

Spain 

25  ->  75 

U.K. 

42  ->  71 

Germany 

28  ->  51 

Argentina 

15 ->  47 

Canada 

18 ->  43 

Malaysia 

15 ->  42 

Mexico 

<10 ->  39 

Australia 

23  ->  29 

Egypt 

10 ->  27 

Chile 

<10 ->  20 

Philippines 

14 ->  20 

Colombia 

<10 ->  18 

and  Italy,  Israel,  Singapore, 
Hong  Kong,  and  Pakistan 

1001  to  2000 


301  to  500 

201  to  2000+ 

29%  1  to  100 

51% 

101  to  200 
19.8% 


•  SCAMPI  A  reports  from  60  countries 

•  72%  of  adopters  are  commercial  orgs 

•  2/3  Services;  1/5  Manufacturing 

•  Approx.  70%  of  adopters  in  US  are 
contractors  for  military/gov’t  or  are  gov’t 


Est’d  1,100+K  work  in 
orgs  that  have  had  a 
SCAMPI  A  appraisal. 


Wholesale  Trade 


Transportation, 

Communication,  Electric,  Gas 
and  Sanitary  Services 

3.2%  Finance,  Insurance  and  Real 

5.9% 


Services 

66% 


Manufacturing 
19% 


Public  Administration 


_ Primary  Metal  Industries 

1.0% 

Industrial  Machinery  And 
' —  Equipment 

2.2% 

Electronic  &  Other  Electric 
^ —  Equipment 

5.1% 

Instruments  And  Related 
—  Products 

5.9% 


Engineering  &  Management 
Services 
13.3% 


Other  Services 


http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 

Is  the  source  for  these  statistical  analyses. _ 


Software  Engineering  Institute  CarnegieMellon 


CMMI  V1.3 
October  6,  2009 

©2009  Carnegie  Mellon  University 


Countries  Where  Appraisals  Have  Been 
Performed  and  Reported  to  the  SEI 


.  \  X  X 

1  -  -->  X-  \  X  ^ 

■XX 

X\  /  A 

^ 

HHHHB 

Argentina 

Australia 

Austria 

Bahrain 

Bangladesh 

Belarus 

Belgium 

Brazil 

Bulgaria 

Canada 

Chile 

China 

Colombia 

Costa  Rica 

Czech  Republic 

Denmark 

Dominican  Republic 

Egypt 

Finland 

France 

Germany 

Greece 

Hong  Kong 

Hungary 

India 

Indonesia 

Ireland 

Israel 

Italy 

Japan 

Korea,  Republic  Of 

Latvia 

Lithuania 

Luxembourg 

Malaysia 

Mauritius 

Mexico 

Morocco 

Nepal 

Netherlands 

New  Zealand 

Norway 

Pakistan 

Panama 

Peru 

Philippines 

Poland 

Portugal 

Romania 

Russia 

Saudi  Arabia 

Singapore 

Slovakia 

South  Africa 

Spain 

Sri  Lanka 

Sweden 

Switzerland 

Taiwan 

Thailand 

Turkey 

Ukraine 

United  Arab  Emirates 

United  Kingdom 

United  States 

Uruguay 

Viet  Nam 
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Number  of  Appraisals  and  Maturity  Levels 
Reported  to  the  SEI  by  Country 


Country 

Number  of 
Appraisals 

Maturity 
Level  1 
Reported 

Maturity 
Level  2 
Reported 

Maturity 
Level  3 
Reported 

Maturity 
Level  4 
Reported 

Maturity 
Level  5 
Reported 

Country 

Number  of 
Appraisals 

Maturity 
Level  1 
Reported 

Maturity 
Level  2 
Reported 

Maturity 
Level  3 
Reported 

Maturity 
Level  4 
Reported 

Maturity 
Level  5 
Reported 

Argentina 

64 

45 

12 

2 

3 

Malaysia 

56 

20 

31 

5 

Australia 

32 

1 

7 

5 

2 

4 

Mauritius 

1 0  or  fewer 

Austria 

1 0  or  fewer 

Mexico 

57 

24 

25 

3 

4 

Bahrain 

1 0  or  fewer 

Morocco 

10  or  fewer 

Bangladesh 

1 0  or  fewer 

Nepal 

1 0  or  fewer 

Belarus 

1 0  or  fewer 

Netherlands 

1 0  or  fewer 

Belgium 

1 0  or  fewer 

New  Zealand 

10  or  fewer 

Brazil 

106 

1 

50 

42 

1 

9 

Norway 

1 0  or  fewer 

Bulgaria 

1 0  or  fewer 

Pakistan 

25 

1 

18 

4 

1 

Canada 

51 

1 

12 

22 

5 

3 

Panama 

10  or  fewer 

Chile 

30 

17 

10 

2 

Peru 

1 0  or  fewer 

China 

745 

1 

117 

540 

27 

41 

Philippines 

21 

2 

11 

7 

Colombia 

22 

7 

11 

1 

2 

Poland 

10  or  fewer 

Costa  Rica 

1 0  or  fewer 

Portugal 

1 0  or  fewer 

Czech  Republic 

1 0  or  fewer 

Romania 

10  or  fewer 

Denmark 

1 0  or  fewer 

Russia 

1 0  or  fewer 

Dominican  Republic 

1 0  or  fewer 

Saudi  Arabia 

1 0  or  fewer 

Egypt 

34 

1 

17 

11 

2 

2 

Singapore 

19 

3 

10 

1 

4 

Finland 

1 0  or  fewer 

Slovakia 

1 0  or  fewer 

France 

141 

4 

81 

45 

1 

2 

South  Africa 

1 0  or  fewer 

Germany 

64 

9 

32 

11 

1 

1 

Spain 

105 

1 

60 

35 

2 

4 

Greece 

1 0  or  fewer 

Sri  Lanka 

1 0  or  fewer 

Hong  Kong 

18 

2 

11 

5 

Sweden 

1 0  or  fewer 

Hungary 

1 0  or  fewer 

Switzerland 

1 0  or  fewer 

India 

409 

14 

191 

24 

166 

Taiwan 

117 

1 

74 

38 

2 

Indonesia 

1 0  or  fewer 

Thailand 

27 

12 

13 

1 

Ireland 

1 0  or  fewer 

Turkey 

14 

12 

2 

Israel 

17 

3 

10 

2 

Ukraine 

1 0  or  fewer 

Italy 

31 

14 

14 

United  Arab  Emirates 

1 0  or  fewer 

Japan 

267 

17 

75 

121 

13 

16 

United  Kingdom 

93 

3 

42 

30 

1 

3 

Korea,  Republic  Of 

138 

1 

47 

61 

13 

7 

United  States 

1272 

27 

448 

462 

21 

124 

Latvia 

1 0  or  fewer 

Uruguay 

10  or  fewer 

Lithuania 

1 0  or  fewer 

Viet  Nam 

12 

9 

1 

2 

Luxembourg 

1 0  or  fewer 
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Three  Complementary  Constellations 


CMMI-DEV  provides 

guidance  for 
measuring,  monitoring, 
and  managing 
development  processes 


CMMI-SVC  provides 
guidance  for  those 
providing  services 
within  organizations  and 
to  external  customers 


CMMI-ACQ  provides 

guidance  to  enable 
informed  and  decisive 
acquisition  leadership 
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CMMI-ACQ  V1.2 
Acquisition  Process  Areas 


Solicitation  & 

Supplier 

Agreement 

Agreement 

yr  Management 

Development 

X 

Acquisition 
Requirements  ^ 
Development 


Acquisition 

Technicai 

Management 


1 6  Core  Process 
Areas 


Acquisition 

Validation 


Acquisition 

Verification 
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Visibility  into  the  Team’s  Capability 


Operational 

Need 


Acquirer 


Developer 
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CMMI-SVC  V1.2 


Capacity  and 
Availability 
Management 


Incident 
Resolution  & 
Prevention 


/  16  Core  \ 

Service 

.  .  /  Process  Areas  I  .  . 

Service 

Continuity 

\  and  1  Shared  / 

Delivery 

PA  (SAM) 


PA  Addition 


Service 

System 

Development 


Strategic 

Service 

Management 


Service 

System 

Transition 
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People  CMM  Primary  Objective 


The  primary  objective  of  CMMI  (DEV,  ACQ,  SVC)  is  to  improve  the 
capability  of  an  organization’s  processes  within  specific  domains. 

The  primary  objective  of  the  People  CMM  is  to  improve  the  capability  of  an 
organization’s  workforce  through  enhanced  management  and  human 
capital  processes. 


(The  People  CMM  defines  capability  as  the  level  of  knowledge,  skills, 
and  process  abilities  available  within  each  workforce  competency  of  the 
organization  to  build  its  products  or  deliver  its  services.) 


CMMI  Planned  Elements  -  Multi-Model  (1) 


Improving  the 
interfaces  is  of 
interest  to  both 
government  and 
industry.... 
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CMMI  Planned  Elements  -  Multi-Model  (2) 


Multiple  models  complicate  process  improvement  -  but  make  it  much  more 
powerful  by  addressing  specific  needs  in  various  environments.... 


Enterprise 

oriented 


/ 


Discipiine/ 
domain  specific 


Governance 

(including  external  mandates,  regulations, 
and  internally  chosen  governance) 


EFQM 


Lean  FDA/51  OK 
eSCM-cfL 
Six  Sigma  SOX  / 
eSGM-SP  eOBFF  -iSO-900e^/ - 


/ 


Organizationai  infrastructure 
and  readiness 

(including  business,  engineering,  and 
change/improvement  practices) 


Tacticai 

(procedural,  for  both  improvement 
and  engineering  tasks) 


P-CMM 


/ 


SCOR 


/ 


/ 


ISO  12207 
ITIL 


CMMI 

SWEBOK 


- GjSHM - PSM- - 


6S/DMAIC  /  IDEAL 

6S/DFSS  /  RUP  TSP 

/ 


ATAM 


Agile 


Q. 

3 

O 

L_ 

O) 

(/) 

(/) 

CD 

O 

O 


O 


3 

CD 

C 

g 

g 

o 

0 

■o 

O) 

c 

C/) 

CD 

2 

o 

c 


Increasing  decision  authority  of  process  group 
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Planned  Sequence  of  Models 
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Schedule  for  CMMI  V1.3  Models 


Preparation  CPs  QA 

Jan  -  May  2009 _ June  -  Oct  2009  July  -  Nov  1, 2010 


CCB  Review  of  CPs  CCB  Review  of  Redlines 
July -Oct  2009  Nov  2009 -April  2010 


CCB  Review  of  VI  .3s 
July  2010 

D 


Redlines 

Aug  2009  -  April  2010 


VI  .3s 

May  -  July  2010 


Piloting  P-Drafts* 
Nov,  Jan,  and  June 


D 


D 


D 


Entire  Project  =  Jan  2009  to  November  1 , 2010 


*  Piloting  will  include  candidate  solutions  for  appraising  multiple 
constellations  as  well  as  a  training  approach  for  CMMI. 
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CMMI  V1.3  Criteria 


Correct  identified  model,  training  material,  or  appraisal  method  defects  or  provide  enhancements. 
Incorporate  amplifications  and  clarifications  as  needed. 

Accommodate  potential  additions  to  model  coverage  (e.g.,  safety,  security,  life  cycle)  only  by  specific 
direction  of  the  CMMI  Steering  Group. 

Decrease  overall  model  size  in  v1 .3  if  possible;  increases,  if  any,  must  not  be  greater  than  absolutely 
necessary. 

Model  and  method  changes  should  avoid  adversely  impacting  the  legacy  investment  of  adopting 
companies  and  organizations. 

Changes  to  model  architecture  will  only  be  incorporated  with  specific  CMMI  Steering  Group  authorization. 
Changes  may  only  be  initiated  by  Change  Requests  or  the  CMMI  Steering  Group. 

Editorial  changes  to  training  may  be  released  in  advance  of  v1 .3. 

Changes  must  not  cause  retraining  of  the  nearly  100,000  (as  of  Dec  2008)  personnel  already  trained  in 
CMMI.  Upgrade  training  may  be  needed,  especially  for  Instructors,  Lead  Appraisers,  and  appraisal  team 
members. 
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CMMI  Product  Suite,  Version  1.3 


Version  1 .3  will  focus  on  but  not  be  limited  to  the  following: 

•  High  Maturity 

•  Appraisal  efficiency 

•  Consistency  across  constellations 

•  Simplify  the  generic  practices 

Version  1.3  is  change  request  (CR)  driven.  Events  such  as  this  conference 
presentation  are  for  information  sharing  and  dialogue. 
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What  Have  We  Missed? 
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